Appearance
Incidents and Judgment
446. The image registry becomes unavailable during a production rollout; what happens to existing Pods and new Pods, and how would you reduce the risk?
The short answer
Running pods keep running, because their images are already on their nodes. New pods start only if their node already has the exact image and the pull policy allows using it, so pods placed on fresh nodes, or using imagePullPolicy: Always, fail with ImagePullBackOff. A rolling update with maxUnavailable: 0 simply stalls, keeping the old pods serving. Reduce the risk with a highly available or geo-replicated registry, a pull-through cache, deploying by digest with IfNotPresent, and alerts on pull failures.
What happens, case by case:
| Situation | Result |
|---|---|
| Pods already running | Unaffected |
Pod restarts on the same node, image cached, IfNotPresent | Starts normally |
imagePullPolicy: Always | Fails: it always asks the registry, even if the image is cached (question 353) |
| New node from autoscaling, or a replacement for a failed node | Fails to pull |
Rolling update with maxUnavailable: 0 | New pods stall; old pods keep serving; the rollout waits |
Rolling update with high maxUnavailable | Old pods removed, new ones cannot start: capacity drops |
The hidden danger is not the rollout, which stalls safely. It is a node failure or a traffic spike during the outage: new pods cannot start anywhere new, so the cluster cannot heal or scale.
Reducing the risk:
- A highly available registry, such as a zone-redundant registry tier, or geo-replication to each region you run in (question 135).
- A pull-through cache or mirror inside your network, so nodes pull from nearby and keep working when the upstream is down.
- Deploy by digest with
IfNotPresent. A digest never changes, so a cached copy is always correct, and no registry call is needed. - No dependency on public registries at runtime: copy base and third-party images into your own registry (question 477).
- Retention rules that never delete deployed images, so a rollback can always pull.
- Alert on pull errors, and pause automatic rollouts when the registry is unhealthy.
447. How would you deploy a service using canary traffic while monitoring errors and latency before full rollout?
The short answer
Run the new version next to the stable one and send it a small share of traffic, such as 5%. A rollout controller, such as Argo Rollouts or Flagger, compares the canary's error rate and latency with set thresholds, using queries against your metrics, for a few minutes at each step. If they pass, it raises the share step by step to 100%. If any check fails, it moves all traffic back to the stable version and stops, with no human needed.
An Argo Rollouts canary for checkout:
yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout
namespace: shop
spec:
replicas: 10
selector:
matchLabels: { app: checkout }
template:
metadata:
labels: { app: checkout }
spec:
containers:
- name: app
image: ghcr.io/shop/checkout@sha256:9f2c...
strategy:
canary:
canaryMetadata:
labels: { track: canary } # (1)
analysis:
templates:
- templateName: checkout-health # (2)
startingStep: 1
steps:
- setWeight: 5 # (3)
- pause: { duration: 10m }
- setWeight: 25
- pause: { duration: 10m }
- setWeight: 50
- pause: { duration: 10m } # (4)
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: checkout-health
namespace: shop
spec:
metrics:
- name: error-rate
interval: 1m
failureLimit: 2 # (5)
successCondition: result[0] < 0.005
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{app="checkout",track="canary",status=~"5.."}[2m]))
/ sum(rate(http_requests_total{app="checkout",track="canary"}[2m]))
- name: p99-latency
interval: 1m
failureLimit: 2
successCondition: result[0] < 0.8
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
histogram_quantile(0.99, sum by (le) (
rate(http_request_duration_seconds_bucket{app="checkout",track="canary"}[2m])))- Label canary pods
track: canary, so metrics can be split between canary and stable. - Background analysis runs the checks continuously from step 1 onwards.
- 5% first. Without a traffic router, Argo approximates weights with pod counts; with a gateway or service mesh, the split is exact.
- Each step waits while analysis keeps checking. After the last step, the rollout goes to 100%.
- Two failed checks abort the rollout: traffic returns to stable, and the canary is scaled down.
The two queries are the error-rate and p99 formulas from question 626, filtered to canary pods.
What makes canaries trustworthy:
- Enough traffic in the canary for the numbers to mean something. At 5% of 10 requests per second, ten minutes is only 300 requests.
- Compare with stable, not only with fixed numbers, so normal daily patterns do not cause false alarms (question 283).
- Business signals too, such as completed checkouts, not only technical ones (question 293).
- Sticky sessions where needed, so one user does not flip between versions on every request.
448. How would you design a safe rollback when application configuration and container image versions change together?
The short answer
Version the configuration together with the image, so one release is one unit: image digest plus configuration, stored in Git and deployed together. Give each configuration version its own name, for example a ConfigMap with a content hash in its name. Then a new version creates new pods that point at the new configuration, while the old pods keep the old one. Rolling back then means returning to the previous unit, through a Helm rollback or a Git revert, which restores both at once.
The problem with in-place configuration. The release changes the image to version 2 and edits the checkout-config ConfigMap. Version 2 fails, and you roll back the Deployment to version 1. But the ConfigMap is still the new one, which version 1 may not understand. Now the rollback fails too.
The fix: name configuration by its content. Kustomize's configMapGenerator adds a hash of the content to the name:
yaml
configMapGenerator:
- name: checkout-config
literals:
- PAYMENT_TIMEOUT_MS=3000
- CURRENCY_DEFAULT=EURThis produces a ConfigMap such as checkout-config-7b9t2m6k4f, and rewrites the Deployment to point at that exact name. Changing a value produces a new name, which changes the pod template, which starts a normal rolling update. The old ConfigMap still exists, so old pods, and a rollback, keep using the old values (question 380).
The release unit and its rollback:
| Tool | One release is | Rollback |
|---|---|---|
| Helm | A revision: rendered manifests including config and image | helm rollback checkout 41 (question 505) |
| GitOps | A commit in the environment repository | Revert the commit; Argo CD or Flux sync it (question 512) |
| Kustomize with hashed ConfigMaps | The rendered output of one commit | Apply the previous commit |
Also make configuration backward compatible, like database changes: new settings get defaults, so the old version ignores them and the new version works without them. Then image and configuration can even roll back separately when needed. Secrets follow the same idea with versioned secret names or versions in the secret store.
449. How would you investigate a release where the pipeline says the new commit was deployed but the running Pods use an older image digest?
The short answer
Compare what the pipeline changed with what the cluster is running, step by step. Check that it changed the right cluster, namespace and Deployment. Check whether the Deployment's spec has the new image but its rollout is stuck, so old pods are still serving. Check whether something reverted the change: a GitOps controller syncing an older Git state, or another pipeline or Helm release managing the same object. And check tag problems: a reused tag with IfNotPresent can leave nodes running an old image under a new-looking name.
Step by step:
- What does the Deployment ask for?
bash
kubectl -n shop get deploy checkout -o jsonpath='{.spec.template.spec.containers[0].image}'If it is the old image, the change never landed, or was reverted. If it is the new one, the rollout is the problem.
- Right target? The pipeline's kube context, cluster, namespace and resource name. Pipelines that deploy to the wrong cluster report success happily (question 299).
- Is the rollout stuck?
bash
kubectl -n shop rollout status deploy/checkout
kubectl -n shop get rs -l app=checkoutA new ReplicaSet with zero ready pods means new pods fail to start, and old pods keep serving (questions 434 and 433).
- Was it reverted? In GitOps, Argo CD or Flux applies what Git says. A pipeline that ran
kubectl set imagedirectly gets undone at the next sync, because Git still has the old digest. Check the application's sync history (question 508). - Two owners? A Helm release and a
kubectl applypipeline both managingcheckoutfight each other. ThemanagedFieldsin the object show who last changed which field. - Mutable tags. If the image is
checkout:latestor a reused version tag withimagePullPolicy: IfNotPresent, nodes that cached an older image under that tag keep using it (question 353). The fix is deploying by digest. - Admission webhooks can rewrite images, for example to a mirror registry. Compare the spec with the pod's
imageID.
Prevent it: deploy by digest, have one owner per object, use GitOps as the only path, and make the pipeline's final check compare the digest running in pods with the digest it built (question 432).
450. How would you design a release process that supports auditability, least privilege, artifact signing, and a tested rollback path?
The short answer
Make Git the record of every change and the pipeline the only way to production. Each release is a reviewed pull request, built once, signed, with an SBOM and provenance. Production changes run under short-lived, narrowly scoped identities, with human approval recorded on the environment. The cluster only accepts signed images from the trusted pipeline. Rollback is the same path backwards, a revert or a previous signed digest, and it is practised regularly so it is known to work.
| Need | How |
|---|---|
| Who changed what, and why | Pull requests with reviews; commit messages linking tickets (question 196) |
| What was deployed, when, by whom | Deployment records per environment: digest, commit, approver, time |
| Least privilege | OIDC identities per environment; people read-only in production (question 271) |
| Only trusted artifacts | Signing and provenance in the pipeline; verification in the cluster (question 275) |
| Approvals | Environment protection rules; approvals stored by the platform (question 257) |
| Rollback | Previous signed digest, or a Git revert; same pipeline, same checks |
| Proof rollback works | Rollback drills in staging and occasionally in production |
| Tamper-evident history | Audit logs from Git, CI, cloud and cluster sent to central storage (question 555) |
A release, end to end:
- A developer opens a pull request; checks and a reviewer approve it.
- Merge builds one image, signs it, and attaches an SBOM and provenance.
- The pipeline writes the digest to the staging folder of the GitOps repository; Argo CD deploys it.
- After staging checks pass, a promotion pull request copies the digest to production, and a second person approves.
- The production cluster verifies the signature at admission, and the canary analysis decides the rollout.
- Every step leaves a record that links commit, digest, approvers and time.
Rollback uses the same path. Reverting the production commit deploys the previous digest, which is already signed and verified. No special "emergency" route skips the checks, so emergencies do not become security holes (question 298).
Keep it practised. A rollback that has not been run in months may fail on something nobody noticed, such as a deleted old image or an incompatible database change (question 444).
451. A pipeline passes all tests, but a Kubernetes readiness probe repeatedly fails after deployment; how do you determine whether the probe or the application is wrong?
The short answer
Ask the app directly, from inside the pod, exactly what the probe asks: the same port, path, headers and timeout. If the app answers quickly and correctly there, the probe definition is wrong, such as the wrong port, the wrong path, or a timeout that is too short. If the app does not answer, or answers with an error, the app is wrong: read its logs for the reason, such as a dependency it cannot reach or start-up work not finished. The pod's events show the probe's exact failure message.
Step by step:
- Read the failure message:
bash
kubectl -n shop describe pod checkout-7d9f-x2 | grep -A3 -i readinessMessages such as connection refused, context deadline exceeded or HTTP probe failed with statuscode: 503 say different things.
- Compare the probe with the app:
yaml
readinessProbe:
httpGet:
path: /health/ready
port: 8080
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 3Is the app really listening on 8080, and does /health/ready exist in this version?
- Ask the app the same question from inside the pod, or from a debug container if the image has no tools (question 414):
bash
kubectl -n shop exec checkout-7d9f-x2 -- wget -qO- -T 1 http://127.0.0.1:8080/health/ready| Result | Points to |
|---|---|
Quick 200 here, probe still fails | The probe: wrong port name, wrong path, or the app only listens on 127.0.0.1 while the kubelet calls the pod IP |
connection refused | The app is not listening on that port: wrong port, or it has not started yet |
| Slow answer, over 1 second | Timeout too short, or the endpoint does heavy work |
503 with a reason | The app says it is not ready: read its logs for why |
- Read the app's logs for the reason it reports itself not ready, such as "cannot connect to Redis", which may be a network policy, a missing Secret, or a new dependency the cluster lacks.
Why the pipeline passed. Its checks ran without the cluster's real network, secrets, policies and dependencies. Readiness is exactly where those differences show up, which is why it is valuable (question 436).
452. How would you diagnose a service that fails only under load because of CPU throttling, memory limits, or insufficient replicas?
The short answer
Reproduce the load in staging or watch production during a peak, and look at four things together: CPU throttling, memory against limits, replica count against the autoscaler's maximum, and the service's latency and errors. Throttling shows as latency rising while CPU use sits flat at the limit. Memory problems show as OOMKilled restarts. Too few replicas shows as the autoscaler at its maximum, or reacting too slowly, while each pod is saturated. Fix the specific cause, then raise the load again to confirm.
What each cause looks like:
| Cause | Signals | Fix |
|---|---|---|
| CPU throttling | Throttled-period ratio high; CPU flat at the limit; p99 latency up | Raise or remove the CPU limit, or right-size requests (question 392) |
| Memory limit | OOMKilled, restarts under load, memory climbing to the limit | Raise the limit after checking it is not a leak (question 411) |
| Not enough replicas | HPA at max replicas, or scaling too slowly; every pod busy | Raise max, scale on a better metric, start scaling earlier (question 398) |
| No room for new pods | New pods Pending during the spike | Cluster autoscaler limits, node pool size (question 399) |
| Downstream limit | Database connections exhausted; dependency latency up | Pool sizes, caching, protecting the dependency (question 582) |
The throttling query:
text
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="shop",container="app"}[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="shop",container="app"}[5m]))It gives, per pod, the share of CPU time slices in which the container wanted more CPU than its limit allowed. Values above about 0.25 usually mean latency is suffering.
Reproducing it. Use a load generator, such as k6, against staging with the same limits and replica settings as production, and raise the rate step by step while watching the dashboard. The first signal to bend tells you the bottleneck.
Common traps:
- HPA on CPU with throttled pods. A throttled pod cannot show more than its limit, so CPU utilisation looks capped and scaling may lag. Scaling on request rate or latency can work better (question 397).
- Slow start-up. New pods that take a minute to become ready arrive too late for a sudden spike. Keep more headroom, or scale on a schedule before known peaks.
453. How would you design separate development, test, staging, and production environments while minimizing configuration drift?
The short answer
Build every environment from the same code: the same infrastructure modules and the same Kubernetes manifests, with a small, explicit set of differences per environment, such as size and addresses. Promote the same artifacts, the same image digests, from one environment to the next. Make Git and the pipeline the only way to change any environment, detect drift automatically, and keep differences visible and few, so staging behaves like production in every way that matters.
One base, small differences:
text
deploy/
base/ the manifests for every environment
overlays/
dev/ 1 replica, small requests, debug logging
test/ 2 replicas
staging/ 3 replicas, production-like data shape
prod/ 6 replicas across 3 zones, strict policiesEach overlay only patches what truly differs. A reviewer can read the full list of differences in a few files (question 506).
The same idea for infrastructure: one Terraform module per component, called from each environment's folder with its own settings and its own state (question 602).
Rules that keep environments close:
| Rule | Prevents |
|---|---|
| Promote digests, never rebuild per environment | "It worked in staging" with a different build (question 134) |
| Same versions of Kubernetes, add-ons and managed services | Surprises from version differences |
| Same policies, in audit mode in dev if needed | A deployment that only fails in production |
| No manual changes; read-only access for people | Hand edits that nobody records (question 600) |
| Scheduled drift detection | Slow, silent divergence |
| Differences listed in one place | Hidden, forgotten differences |
Where differences are fine: scale, cost-saving choices such as smaller databases, data, and secrets. Where they are dangerous: networking rules, security policies, versions, and the deployment process itself.
Short-lived environments, created per pull request from the same code and deleted after, help teams check changes in isolation without keeping many long-lived copies to drift.
454. How would you handle a compromised deployment credential and verify that no unauthorized workloads or images were deployed?
The short answer
Revoke the credential immediately and replace it, then investigate. Use audit logs from the cloud, the Kubernetes API, the registry and the CI system to list everything that credential did since it might have leaked. Compare what is running in every cluster with what your release records say should be running, digest by digest, and look for anything new: workloads, CronJobs, DaemonSets, RBAC bindings, webhooks and images. Remove what should not exist, and close the gap that let the credential leak.
In order:
- Revoke and rotate. Disable the credential, such as a service account key or kubeconfig token. Rotate anything it could read, because those secrets may be stolen too (question 267).
- Scope the window. When could it have leaked? Use the earliest possible time, not the time you noticed.
- Read the audit logs for that identity: Kubernetes API audit logs, cloud activity logs, registry push logs, CI logs (question 555).
- Compare running with expected:
bash
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | sort -uEvery image should match a digest from your signed release records. Anything else is suspicious.
- Look for persistence, the things an attacker adds to keep access:
| Look for | Why |
|---|---|
| New ClusterRoleBindings or RoleBindings | Granting themselves lasting rights |
| New service accounts and tokens | A credential that survives your rotation |
| DaemonSets, CronJobs | Code that runs on every node, or keeps coming back |
| Mutating admission webhooks | Silently changing future deployments |
| Images pushed outside the pipeline | Poisoned tags waiting to be deployed |
| Cloud resources: new users, keys, machines | Access and crypto mining beyond the cluster (question 558) |
- Remove and redeploy from known-good sources: Git and signed digests.
- Fix the cause and add controls: short-lived federated credentials instead of keys, admission policies that only accept signed images, and alerts on unexpected RBAC changes (questions 308 and 500).
455. What would you include in a post-incident review after a failed release, and how would you turn the findings into automated prevention?
The short answer
Include a factual timeline and the impact on users and the business. Say how the problem was detected and how long each stage took. List the contributing causes across code, process and tooling, and what helped and what hurt. End with specific actions, each with an owner and a date. Keep it blameless. Then turn each lesson into something automatic where possible: a pipeline check, an admission policy, an alert, a canary analysis rule or a safer default, so the protection does not depend on people remembering.
What the review contains:
- Summary: what happened, in three sentences.
- Impact: users affected, duration, failed orders, error budget used (question 631).
- Timeline: release time, first errors, alert, acknowledgement, diagnosis, rollback, recovery (question 638).
- Contributing causes: usually several, such as "the config key was renamed", "staging used a default value", and "the canary did not check checkout success".
- What went well: such as "rollback took 2 minutes".
- What made it worse: such as "the alert went to an old channel".
- Actions, each with an owner and a due date.
From finding to automation:
| Finding | Automated prevention |
|---|---|
| A renamed config key broke start-up | Start-up validation of required settings; the pod fails readiness, not users |
| Staging had a default production lacked | Same overlays, with a check that lists differences |
| The canary missed broken checkouts | Add checkout success rate to the canary analysis (question 447) |
| An unsigned image reached production | Admission policy rejecting unsigned images (question 500) |
| Detection took 25 minutes | SLO burn-rate alert on the checkout journey (question 631) |
| Rollback was slow and manual | Automated rollback on failed rollout in the pipeline (question 431) |
Follow through. Track actions like any other work, review them in a regular meeting, and close the review only when they are done. A review whose actions never happen is only a story.
456. How would you prioritize the first five checks when production is down immediately after a release?
The short answer
First, confirm the scope from the user-facing signals: what is broken and for whom. Second, since it started right after a release, roll back or turn off the change straight away, without waiting for the root cause. Third, check whether the rollback restored service. Fourth, if it did not, look at what else changed at the same time, such as configuration, infrastructure or a dependency, and the health of shared parts. Fifth, communicate and gather evidence for the review.
Each step, in a little more detail:
- Scope. The SLO dashboard and error rate by route: is it everything, one journey, one region? This takes a minute and guides everything else (question 629).
- Roll back. A release just went out and production broke: the odds strongly favour the release. Roll back, or turn off its feature flag, before debugging (question 427). Do not wait for proof.
- Recovered? Watch the same signals for a few minutes. If they return to normal, service is restored, and the root cause can wait for working hours.
- Look wider, if not. Something else changed too, or the release changed something a rollback does not undo, such as a database migration or a configuration change outside the image. Check recent changes of every kind, shared dependencies, and the platform itself (question 423).
- Communicate and record. Declare the incident, post updates on a schedule, and keep a timeline of every action. Evidence gathered now makes the review honest later (question 637).
What not to do first: read code, debug in production, or deploy a "quick fix" forward before trying the rollback. A forward fix under pressure is how one incident becomes two.
457. How would you explain the trade-offs of faster releases, stricter quality gates, and operational risk to a product manager?
The short answer
Explain that speed and safety are not opposites: years of DORA research show the best teams release more often and fail less, because small, frequent changes are easier to check, easier to roll back and less risky each. Strict gates help when they are fast and automatic, and hurt when they are slow and manual, because they push teams to batch changes. Use shared numbers, such as the SLO and error budget, to decide together: while the budget is healthy, ship faster; when it is spent, slow down and invest in reliability.
Start from what they care about: features reaching customers, and customers not being hurt. Then explain with numbers, not opinions.
Three points that usually land:
- Small batches are safer. A release with one change that breaks is fixed in minutes, because you know what changed. A release with fifty changes that breaks takes hours to untangle.
- Gates have a cost. A manual approval that takes two days means work piles up, and the next release is bigger and riskier. An automatic check that takes two minutes is nearly free (question 290).
- Risk can be priced. "An hour of checkout downtime costs about £40,000" turns reliability into a business decision the product manager can weigh against features.
A shared rule, so it is not a negotiation each time:
| Error budget | What it means for delivery |
|---|---|
| Healthy | Ship as fast as the pipeline allows |
| Getting low | Smaller changes, more canary time |
| Spent | Feature releases pause; fixes and reliability work first (question 631) |
Show the measures together, both speed and stability: deployment frequency and lead time next to change failure rate and recovery time. If speed improves and failures do not rise, the approach is working (question 294).
Offer options, not refusals. Instead of "we cannot release on Friday", say "we can release on Friday behind a feature flag, turned on for 5% of users, with automatic rollback".
458. How would you keep the deployment system simple enough to operate while still meeting security, availability, and audit requirements?
The short answer
Offer one well-built path that every team uses, instead of many custom ones, and build it from managed services and widely used tools rather than home-made ones. Put security, audit and availability inside that path as defaults, such as signing, OIDC, approvals and canaries, so teams get them without extra work. Remove tools and steps that do not earn their keep, document the path, and measure it. Complexity should be added only when a real requirement cannot be met otherwise.
The idea of a paved road. A platform team builds one supported way to go from code to production: a template repository, a shared pipeline, a standard deployment chart, GitOps and default dashboards. Teams can step off it, but then they own the extra work. Most teams stay on it because it is the easiest way (question 305).
Simplicity choices that still meet strict requirements:
| Requirement | Simple way to meet it |
|---|---|
| Audit | Git as the record; pipeline and platform logs kept centrally |
| Least privilege | OIDC federation per environment instead of managing keys |
| Supply-chain security | Signing and SBOMs built into the shared pipeline once |
| Availability | Managed Kubernetes or container services; rolling updates and PDBs by default |
| Approvals | The CI platform's environment protection, not a custom ticket bot |
| Rollback | Git revert or Helm rollback, the same for every team |
Signs the system is getting too complex:
- Nobody can explain the full path from commit to production on a whiteboard.
- Several tools do the same job, such as two deployment tools in one cluster.
- Incidents are often caused by the delivery system itself.
- New team members take weeks to ship their first change.
Rules to keep it small: every new tool must replace something or meet a written requirement; prefer configuration of an existing tool over a new one; and delete unused pipelines, jobs and plugins regularly.
459. If you inherited a fragile pipeline and an unfamiliar Kubernetes cluster, what would you inspect first and why?
The short answer
Learn before changing anything. First make sure you can recover: that access works, backups and the source of truth exist, and you know how to roll back. Then map what exists: which services run, how they get deployed, which versions are in use and whether they are still supported, and what monitoring and alerts exist. Then look at recent pain: incidents, failed runs and manual workarounds. Fix the biggest risks first, such as missing backups or unsupported versions, and make improvements in small, reversible steps.
Week one, in order:
- Access and recovery. Do you have working, personal access with the right rights? Are there backups of anything stateful? Is there a way to recreate the cluster and the pipelines from code? If not, that is the first risk.
- Inventory. What runs in the cluster: namespaces, workloads, add-ons, controllers and their versions.
bash
kubectl version
kubectl get nodes -o wide
kubectl get ns
kubectl get deploy,sts,ds,cronjob -A
helm list -A- Support status. Kubernetes version and its support end date, node images, and add-ons such as the ingress controller, which may be retired (question 491). Unsupported versions block safe upgrades later.
- How things get deployed. Every pipeline that touches the cluster, which identity it uses, and whether there are manual steps or hand-applied resources that exist nowhere in Git (question 600).
- Observability. Which dashboards and alerts exist, who receives them, and whether they fire for real problems.
- Recent pain. Incident history, failed pipeline runs and their causes, and the workarounds people use. Ask the team where it hurts.
- Security basics. Who has cluster-admin, stored long-lived credentials, public endpoints, and images from unknown sources (question 415).
Then improve gradually:
- Write down what you learned, as a map and a list of risks.
- Fix the risks that could cause permanent loss first: backups, single points of failure, credentials.
- Change one thing at a time, with a way back, and measure the effect.
- Avoid rewriting everything at once. The current system, however fragile, works in ways you may not yet understand.
460. What evidence would convince you that the deployment process is reliable rather than merely successful on its last run?
The short answer
Trends over many releases, not one green run. Look at the DORA measures over months: frequent deployments, short lead time, a low change failure rate and fast recovery. Look at the share of pipeline runs that fail for reasons unrelated to the code, such as flaky steps or runner problems. Check that rollbacks and failovers are practised and work, that every service uses the same path, that builds are reproducible and traceable, and that problems are caught by automation before users notice them.
Evidence, and what it shows:
| Evidence | Shows |
|---|---|
| DORA measures over months (question 294) | Speed and stability together, not one good week |
| Change failure rate, trend | How often releases hurt users |
| Time to restore after bad releases | Whether failures are cheap |
| Pipeline failures not caused by code | Flakiness and platform health (question 266) |
| Rollback drills: frequency and duration | Whether rollback is real or only on paper |
| Share of services on the standard path | Whether the process is used, or bypassed |
| Every production digest traceable to a reviewed commit | Control and audit (question 440) |
| Incidents found by alerts, not customers | Detection works (question 638) |
| Review actions completed | The process learns from failures (question 455) |
Signs of a process that only looks reliable:
- Green runs, but people rerun failed jobs until they pass.
- Successful deployments, followed by quiet manual fixes in production.
- Nobody has rolled back in a year, so nobody knows whether it works.
- A few experts deploy the important services by hand "to be safe".
- Long gaps between releases, which hide problems instead of removing them.
The strongest single sign: releases are boring. Teams deploy during working hours, on any day, without a meeting, and when something goes wrong, it is caught early and undone in minutes.