Appearance
Delivery Scenarios
431. Design a complete delivery flow that builds a Docker image, tests it, scans it, publishes it, deploys it to Kubernetes, verifies health, and rolls it back if needed.
The short answer
On every pull request, build the image, run the checks and the test suite, and scan it. On merge, build once, push it to the registry, and record its digest, with an SBOM and a signature. Deploy that exact digest to staging, wait for the rollout to become healthy, and run smoke checks. Promote the same digest to production behind an approval, ideally as a canary, and watch error rate and latency. If the rollout does not become healthy, or the signals get worse, roll back automatically to the previous digest.
A trimmed GitHub Actions workflow for the main branch:
yaml
name: deliver-checkout
on:
push:
branches: [main]
permissions:
contents: read
id-token: write # (1)
packages: write
jobs:
build:
runs-on: ubuntu-latest
outputs:
digest: ${{ steps.push.outputs.digest }}
steps:
- uses: actions/checkout@v4
- run: npm ci && npm test # (2)
- uses: docker/setup-buildx-action@v3
- id: push
uses: docker/build-push-action@v6
with:
push: true
tags: ghcr.io/shop/checkout:${{ github.sha }} # (3)
sbom: true
provenance: mode=max
- uses: aquasecurity/trivy-action@0.28.0 # (4)
with:
image-ref: ghcr.io/shop/checkout@${{ steps.push.outputs.digest }}
severity: CRITICAL,HIGH
exit-code: '1'
- uses: sigstore/cosign-installer@v3
- run: cosign sign --yes ghcr.io/shop/checkout@${{ steps.push.outputs.digest }} # (5)
staging:
needs: build
runs-on: ubuntu-latest
environment: staging
steps:
- uses: azure/login@v2 # (6)
with:
client-id: ${{ vars.AZURE_CLIENT_ID }}
tenant-id: ${{ vars.AZURE_TENANT_ID }}
subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
- run: az aks get-credentials -g rg-shop-staging -n aks-shop-staging
- run: |
kubectl -n shop set image deploy/checkout app=ghcr.io/shop/checkout@${{ needs.build.outputs.digest }}
kubectl -n shop rollout status deploy/checkout --timeout=5m || {
kubectl -n shop rollout undo deploy/checkout; exit 1; } # (7)
- run: ./scripts/smoke.sh https://staging.shop.example # (8)
production:
needs: [build, staging]
runs-on: ubuntu-latest
environment: production # (9)
steps:
- run: echo "same steps as staging, with digest ${{ needs.build.outputs.digest }}"id-token: writelets the job ask for an OIDC token, used for keyless signing and for signing in to Azure without stored keys (question 269).- The test suite runs before anything is pushed. Its contents are the app team's business. The pipeline only needs a clear pass or fail.
- Tag with the commit SHA, so every image traces back to its commit. The tag is for people. The digest, produced by the push, is what deploys.
- Scan the pushed image by digest and fail on critical or high findings that have a fix, according to your policy (question 137).
- Sign the digest with Cosign, keyless, so the cluster can later refuse unsigned images (question 275).
- Federated sign-in to Azure, no secrets stored in GitHub.
- Deploy by digest and wait. If the rollout is not healthy within five minutes, undo it and fail the job.
- Smoke checks: a few quick requests through the real entry point, such as loading the home page and adding to a cart.
- The production environment has required reviewers, so a person approves before this job starts (question 246).
What makes the flow safe, beyond the YAML:
- The cluster enforces it too. An admission policy accepts only signed images from your registry, by digest, so even a person with
kubectlcannot skip the pipeline (question 500). - Readiness probes that check real dependencies, so "rollout healthy" means something (question 404).
- A canary in production, with Argo Rollouts or Flagger watching error rate and latency, and stopping by itself (question 283).
- GitOps is a common alternative to
kubectlin the pipeline: the pipeline writes the new digest into the environment's Git repository, and Argo CD applies it (question 510).
432. How would you ensure the exact image tested in staging is the one deployed to production?
The short answer
Deploy by digest, not by tag. Build the image once, record its digest, deploy that digest to staging, and promote the very same digest to production, never rebuilding and never using a tag such as latest that can move. Store the digest with the release record, sign it after it passes staging, and have the production cluster's admission policy accept only images signed by the staging promotion step. Then compare the digest running in production with the release record.
Why tags are not enough. A tag such as checkout:2.4.1 is a pointer that anyone with push rights can move to a different image. A digest, such as sha256:9f2c…, is the hash of the image's content. If one byte changes, the digest changes. So a digest names one exact image, for ever (question 35).
The chain, step by step:
- Build once on merge, push, and capture the digest from the push output.
- Write the digest down: in the pipeline's outputs, in the release notes, and in the GitOps repository for each environment.
- Deploy to staging by digest:
image: ghcr.io/shop/checkout@sha256:9f2c…. - After staging passes, attach proof to the digest, for example a signature or attestation that says "passed staging" (question 275).
- Promote the same line into the production manifests. In GitOps, that is a pull request copying the digest from the staging folder to the production folder (question 510).
- Enforce at the cluster. A Kyverno or Gatekeeper policy in production rejects images without the "passed staging" attestation (question 500).
- Verify afterwards:
bash
kubectl -n shop get pods -l app=checkout \
-o jsonpath='{range .items[*]}{.status.containerStatuses[0].imageID}{"\n"}{end}'Every line should end with the digest from the release record.
What else must stay the same. The image is only part of what runs. Configuration differs by environment on purpose, such as database addresses. Keep it in versioned manifests, so "what ran in staging" is the image digest plus a known set of settings (question 226).
433. A CI pipeline builds an image successfully, but Kubernetes reports ImagePullBackOff; how do you investigate each layer of the problem?
The short answer
Read the exact error in the pod's events first, because it usually names the layer. Then check each layer in order. Is the image name, tag or digest exactly what was pushed? Does the image exist in the registry, for the node's CPU architecture? Can the node authenticate, through its identity or the pull secret? Can the node resolve the registry's name and reach it over the network? And is the registry itself healthy, or rate-limiting? Fix the layer the evidence points to.
Step by step:
- Read the events:
bash
kubectl -n shop describe pod checkout-7d9f-x2 | sed -n '/Events/,$p'Look for the line starting Failed to pull image. Its end names the reason.
- The reference. Compare the image line in the pod with what the pipeline pushed. A tag from the wrong branch, a missing registry prefix, or
ghrc.ioinstead ofghcr.ioall fail here. - The image exists. Look it up in the registry by tag and digest. Check it has a variant for the node's architecture: an image built only for
amd64fails onarm64nodes with "no matching manifest" (question 82). - Authentication. On managed clusters, check the node identity has pull rights, such as
AcrPull(question 617). With pull secrets, check the secret is in the same namespace, referenced by the pod or its service account, and not expired (question 354). - Network. From a debug pod on the same node, resolve the registry name and connect to it. With private endpoints, the name must resolve to a private address (question 538). Firewalls and proxies can block it too.
- Registry health. Rate limits, such as Docker Hub's for anonymous pulls, an outage, or a deleted repository (question 477).
Related but different: ErrImagePull is the first failure. ImagePullBackOff means Kubernetes has failed several times and is now waiting longer between retries. Once the cause is fixed, the next retry succeeds, or delete the pod to retry at once (question 406).
434. A new image is deployed, but every new Pod enters CrashLoopBackOff; what order of checks would you follow?
The short answer
Start with the evidence the crash left behind: the previous container's logs, the exit code and reason in kubectl describe, and recent events. Exit codes split the cause: 1 is usually the app's own error, such as missing configuration, 137 is a kill, often out of memory, 127 is a missing command, and 126 a permission problem. Then compare what changed in this release: image, command, configuration, secrets, probes and limits. If every new pod fails and the old version worked, roll back first and debug after.
The order:
- Stop the damage. If the old pods are still serving, the rolling update has paused on its own, because new pods never become ready (question 351). If users are hurt, roll back now:
bash
kubectl -n shop rollout undo deploy/checkout- Read the last crash's logs:
bash
kubectl -n shop logs deploy/checkout --previous--previous shows the logs of the container that just crashed, not the new attempt that may not have printed anything yet.
- Read the exit code and reason:
bash
kubectl -n shop describe pod checkout-7d9f-x2| Exit code or reason | Usually means |
|---|---|
| 1, or another small number | The app exited with an error: bad config, failed connection |
137, OOMKilled | Killed for using more memory than its limit (question 411) |
| 137, not OOM | Killed by a signal, often a liveness probe failing (question 402) |
| 127 | Command not found: wrong command or entrypoint |
| 126 | Not executable: file permissions |
| 0, restarting anyway | The process finished; a server that exits is a bug, or restartPolicy is wrong |
- Compare with the last good release: image digest,
commandandargs, environment variables, mounted ConfigMaps and Secrets, the probe settings, resource limits, and the security context (question 436). - Check dependencies at start-up. A missing Secret key, a new required setting, a database the new version cannot reach, or a migration not yet run all crash at start (question 444).
- Reproduce locally with the same image and settings if the cause is still unclear (question 164).
Liveness probes can cause the loop. If the app is slow to start and the liveness probe kills it before it is ready, it will never finish starting. Add a startup probe (question 401).
435. The Deployment reports success, but users still receive errors; how would you validate traffic routing, readiness, application health, and the release itself?
The short answer
"Rollout succeeded" only means the new pods passed their readiness probes. Follow a real request from the outside in. Check where the errors come from: the edge, the load balancer, the gateway, or the app. Check the route and Service point to the right pods, through the Service's endpoints. Check readiness actually tests what users need. Check the new version's own errors and logs. And confirm the running pods really are the new release. If errors started with the release, roll back while you investigate.
Step by step, from the outside in:
- What error, from where? A
502or503from the load balancer or gateway means it could not reach a healthy backend. A500with your app's error page means the app itself failed. The response headers and error page usually reveal which layer answered (question 376). - Is it every user or some? Errors only in one region, one route, or for one kind of request point to routing or to one feature.
- Is the route right? Check the HTTPRoute or Ingress sends the host and path to the right Service, and has no leftover rule pointing at an old Service.
- Does the Service reach the new pods?
bash
kubectl -n shop get endpointslices -l kubernetes.io/service-name=checkoutAn empty list means the Service's selector matches no ready pods. A list with old pod addresses means stale routing. A selector that also matches another Deployment's pods means traffic is split between two apps (question 326).
- Does readiness mean anything? A probe on
/that returns200while the database connection is broken lets broken pods receive traffic. A useful readiness check confirms the app can serve real requests (question 404). - The app's own signals. Error rate and logs by version label. A spike only on the new version is the release. A spike on all versions is something else, such as a dependency.
- Is it really the new release? Check the digest running in the pods (question 432) and whether a cache or CDN is still serving old content.
Roll back on evidence. If the error rate rose exactly at the release, roll back first, then continue the investigation without users waiting (question 293).
436. The application works locally in Docker but fails in Kubernetes; how would you compare environment variables, filesystem, networking, security context, and resource settings?
The short answer
Assume the image is the same and the surroundings differ, then compare the surroundings one by one. Print the environment inside the pod and compare with your local run. Check the filesystem: Kubernetes often runs with a read-only root filesystem, a different user, and mounted volumes that hide image files. Check networking: names, ports and network policies differ from Docker's local network. Check the security context: non-root, dropped capabilities. And check resource limits, which may be far tighter than your laptop.
| Area | Locally in Docker | In Kubernetes | How to compare |
|---|---|---|---|
| Environment | .env file, your shell | ConfigMaps, Secrets, env: | kubectl exec then env, against docker run … env |
| Filesystem | Writable, as root | Often read-only root, non-root user | Look for "read-only file system" or "permission denied" in logs |
| Mounted volumes | None, or your folders | Volumes that may hide files in the image | Check volumeMounts paths against where the app expects files |
| Networking | localhost, Compose service names | Service DNS names, network policies | Resolve and connect from inside the pod |
| User and capabilities | Root, all default capabilities | runAsNonRoot, dropped capabilities | Compare the securityContext with docker run --user |
| Resources | Your laptop's CPU and memory | Requests and limits | Look for OOMKilled and throttling |
| Architecture | Often arm64 on laptops | Often amd64 nodes | Check the image has both (question 82) |
| Signals and shutdown | Rarely stopped | Pods stopped on every rollout | Check PID 1 and graceful shutdown (question 103) |
The fastest checks:
- Run locally the way Kubernetes does, to reproduce:
bash
docker run --rm --read-only --tmpfs /tmp --user 10001 \
--cap-drop ALL --memory 256m --cpus 0.5 \
-e DATABASE_URL=postgres://orders-db:5432/orders \
ghcr.io/shop/checkout@sha256:9f2c...If this fails the same way, the cause is one of these restrictions, and you can find which by removing them one at a time.
- Look inside the failing pod, or a debug copy of it (question 414):
bash
kubectl -n shop debug -it pod/checkout-7d9f-x2 --copy-to=checkout-debug --container=app -- shClassic causes: the app writes to its own folder, which is read-only in the cluster. It binds to port 80, which a non-root user cannot do. It connects to localhost:5432, which worked with a local database but means "this pod" in Kubernetes. It reads a config file that a mounted volume now hides.
437. How would you move secrets from a pipeline into a Kubernetes workload without exposing them in image layers or source control?
The short answer
Best is not to move them through the pipeline at all. Keep secrets in a cloud secret store, and let the workload fetch them at runtime with its own workload identity, either directly from the app or through the External Secrets Operator or the Secrets Store CSI Driver. The pipeline then only deploys a reference, such as the secret's name, never its value. If a secret must pass through a pipeline, inject it at deploy time from the CI system's protected secrets, never into the image, logs or Git.
The preferred pattern:
- The secret lives in Azure Key Vault, AWS Secrets Manager or similar (question 553).
- The pod's service account is linked to a cloud identity with read access to that one secret (question 615).
- An
ExternalSecretresource, committed to Git, names the secret but contains no value:
yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: checkout-payment
namespace: shop
spec:
refreshInterval: 1h # (1)
secretStoreRef:
name: shop-keyvault
kind: SecretStore
target:
name: checkout-payment # (2)
data:
- secretKey: PAYMENT_API_KEY
remoteRef:
key: payment-api-key # (3)- Refresh every hour, so a rotated secret reaches the cluster without a deploy.
- The Kubernetes Secret the operator creates and keeps updated.
- The name in the key vault. Only the name is in Git.
The pod then reads checkout-payment as an environment variable or a mounted file, like any Secret (question 492).
What must never happen:
| Mistake | Why it leaks |
|---|---|
ARG or ENV with the secret in the Dockerfile | Stored in image history and layers (question 80) |
COPY .env into the image | Anyone who can pull the image can read it |
| A plain Secret manifest in Git | Base64 is encoding, not encryption (question 378) |
echo of the value in a pipeline step | Ends up in logs; masking is not reliable (question 270) |
Protect the Secret inside the cluster too: RBAC so only the right service account can read it, and encryption of Secrets at rest (question 379).
438. How would you design a build-and-deploy pipeline that supports many services without rebuilding every service on every commit?
The short answer
Work out which services a commit actually changed, and build only those. Use path filters or a build tool that understands dependencies between folders, so a change in a shared library rebuilds the services that use it, and nothing else. Share one reusable pipeline template that every service calls, so the steps stay the same. Use build caches so even rebuilt services are fast, and deploy each service independently, with its own version and its own rollback.
Step by step:
- Detect changes. Compare the commit with the last successful build of the main branch, and list changed folders:
bash
git diff --name-only origin/main...HEAD | cut -d/ -f2 | sort -uIn a repository laid out as services/checkout/, services/cart/, this prints the service names that changed.
- Include dependents. If
libs/money/changed, every service that imports it must rebuild. Monorepo tools such as Nx, Turborepo, Bazel or Pants build this dependency graph for you (question 193). - Run one job per changed service with a matrix, each calling the same reusable workflow or template (question 260).
- Cache dependencies and Docker layers per service, keyed on its lockfile and Dockerfile (question 224).
- Deploy independently. Each service gets its own image, tag, digest and rollout. One service failing does not block the others.
A GitHub Actions example with path filters:
yaml
on:
push:
branches: [main]
paths:
- 'services/checkout/**'
- 'libs/money/**' # checkout depends on this
- '.github/workflows/checkout.yml'This works for a handful of services. For dozens, a change-detection step that outputs a matrix of services scales better than one workflow file per service (question 255).
Guard against missed rebuilds. Changes to shared things, such as the base image or the pipeline template itself, should rebuild everything, on a schedule if not immediately, so no service quietly keeps an old base image (question 166).
439. How would you prevent an untrusted pull request from publishing an image or deploying to a production cluster?
The short answer
Treat code in a pull request, especially from a fork, as untrusted: it can change the pipeline files and run anything. So pull-request jobs get no secrets, no cloud credentials, and only read permissions, and they never push to the real registry or deploy. Publishing and deploying happen only from protected branches after review and merge, in jobs whose cloud identity only trusts those branches. Production also requires environment approvals, and the cluster only accepts images signed by the trusted pipeline.
Layers of protection:
| Layer | Control |
|---|---|
| Pipeline trigger | Pull requests run on pull_request, which gets a read-only token and no secrets for forks |
| Dangerous triggers | Never check out and run fork code in pull_request_target or similar privileged events (question 250) |
| Permissions | permissions: contents: read by default in every workflow |
| Cloud identity | The OIDC trust only accepts tokens from refs/heads/main or a protected environment (question 269) |
| Registry | Push rights only for the main-branch build identity |
| Environment | Production requires reviewers and only allows the main branch (question 246) |
| Branch protection | Required reviews, required checks, no direct pushes, CODEOWNERS on pipeline files |
| Cluster | Admission policy: only images signed by the main-branch pipeline (question 275) |
Why the cloud identity's condition matters most. Even if a malicious pull request edits the workflow to request a token, the cloud refuses it, because the token says it came from a pull request, not from main. A typical federated credential subject looks like:
text
repo:shop/checkout:environment:productionOnly jobs running in the production environment, which only main can use, match it.
Self-hosted runners need extra care: a pull request's job can leave something behind on a reused machine. Use fresh, short-lived runners for untrusted code, and never give them network access to production (question 271).
440. How would you make Docker builds reproducible and ensure each production deployment is traceable to a source commit?
The short answer
For reproducibility, pin every input: base images by digest, dependencies through lockfiles installed with commands that honour them, tool versions, and build arguments, and build in CI rather than on laptops. For traceability, tag images with the commit SHA, add standard labels for source and revision, generate provenance attestations that record the commit, workflow and builder, and record the image digest in each deployment. Then you can go from a running pod to the exact commit, and back.
Reproducible: the same inputs give the same image.
- Pin the base image by digest:
FROM node:22-slim@sha256:…(question 86). - Lock dependencies and install them strictly, such as
npm ci, which fails if the lockfile andpackage.jsondisagree. - Pin tool versions in the build, such as the exact Node.js and build tool versions.
- Avoid network surprises during the build, such as
curlof a URL whose content changes. - Set a fixed timestamp for files where tools support it, through
SOURCE_DATE_EPOCH, so two builds produce byte-identical layers (question 85).
Traceable: from pod to commit.
dockerfile
ARG GIT_SHA
LABEL org.opencontainers.image.source="https://github.com/shop/checkout" \
org.opencontainers.image.revision="${GIT_SHA}"These are the standard OCI labels for "where the source lives" and "which commit". Then:
- Tag the image with the commit SHA.
- Generate provenance: a signed statement of which commit, workflow and runner built this digest (question 146).
- Record each deployment: environment, digest, commit, who approved, when (question 196).
- Label the pods with the version too, so dashboards and logs can be filtered by release.
Going backwards during an incident:
bash
kubectl -n shop get pod checkout-7d9f-x2 -o jsonpath='{.status.containerStatuses[0].imageID}'
docker pull ghcr.io/shop/checkout@sha256:9f2c...
docker inspect --format '{{ json .Config.Labels }}' ghcr.io/shop/checkout@sha256:9f2c...The first command prints the running digest. The last two pull that exact image and print its labels, which name the source repository and revision.
441. A security scan discovers a critical vulnerability in an image already deployed to production; what would you do first, and how would you remediate it?
The short answer
First, decide how urgent it really is: is the vulnerable package actually used, is it reachable from outside, is there a known exploit, and is a fixed version available. Find every image and environment that contains it, using SBOMs and the registry's scan results. If it is exploitable now, reduce exposure immediately, for example with a WAF rule, a network policy or turning off the feature. Then rebuild with the fixed base image or dependency, push it through the normal pipeline quickly, and confirm the new scan is clean.
Step by step:
- Triage (question 137):
| Question | Why it matters |
|---|---|
| Is the package in a layer the app actually runs? | A vulnerable tool in a build stage that is not in the final image is no risk |
| Is the vulnerable function reachable? | Code that is never called is lower risk |
| Is it exposed to the internet? | Public services first |
| Is there a known exploit, or is it on a known-exploited list? | Active attacks raise urgency to "today" |
| Is a fix available? | Decides between patching and other protection |
- Find everything affected. Search SBOMs and registry scan results for the package and version across all images, not only this one (question 274).
- Contain, if exploitable now: a WAF rule blocking the attack pattern, a network policy cutting unneeded access, scaling down an optional component, or turning off the affected feature with a flag.
- Fix: update the base image or the dependency, rebuild, and let the normal pipeline scan, sign and deploy it. A fast path for urgent fixes should still keep the checks (question 298).
- Confirm: the new image scans clean for this issue, the running pods use the new digest, and old vulnerable images are no longer deployable.
- Look for signs of use: if it was exploitable and exposed, check logs for attack attempts and involve the security team.
Prevent the next scramble: rebuild images regularly to pick up base image fixes, scan running images continuously and not only at build time, and keep SBOMs so "are we affected?" takes minutes (question 166).
442. A new release introduces high latency but no Pod crashes; what telemetry and rollback criteria would you use?
The short answer
Compare the new version with the old one on the same signals: p95 and p99 latency per route, error rate, and throughput, split by version label. Look inside with traces, to see which step got slower, and with resource metrics such as CPU throttling, memory, and database query times. Decide rollback criteria before the release, such as "p99 above 800 ms, or 20% worse than the previous version, for 10 minutes, rolls back", and let a canary controller apply them automatically.
Signals to compare, old version against new:
| Signal | Shows |
|---|---|
| p95 and p99 latency by route | Which requests got slower (question 629) |
| Error rate and timeouts | Slow requests turning into failures |
| Traces of slow requests | Which span grew: a query, a call, the app's own code (question 634) |
| CPU throttling | The new version needs more CPU than its limit allows (question 392) |
| Memory and garbage collection | More memory work, longer pauses |
| Database query time and count | A new query, a missing index, or N+1 queries |
| Downstream latency | A new or slower dependency call |
An N+1 query problem is when code runs one database query to fetch a list, then one more query for every item in it, so a page with 100 items makes 101 queries instead of 1 or 2.
Rollback criteria, decided in advance:
text
Roll back automatically if, for 10 minutes:
p99 latency of /checkout > 800 ms, or
p99 latency of /checkout > 1.2 × the previous version's p99, or
error rate > 0.5%A relative rule, compared with the old version, catches slowdowns even when the absolute numbers are still inside the SLO.
With a canary, Argo Rollouts or Flagger run these queries against the canary and the stable version side by side, and abort the rollout by themselves (question 283). Without one, a person watches the same dashboards after each step of a rolling update, with the decision already written down, so there is no debate at 2 a.m.
After rolling back: reproduce with a trace of a slow request in staging, fix, and release again.
443. A rolling deployment causes temporary downtime; how would you inspect probes, replica counts, resource capacity, and update settings?
The short answer
A correct rolling update should never drop traffic, so downtime means one of four things went wrong. Readiness may pass before the app can really serve, or the app may not shut down gracefully, so requests hit pods that are starting or stopping. There may be too few replicas, or maxUnavailable lets too many go at once. Or the cluster had no room for new pods, so old ones were removed before new ones were ready. Check each, and fix the specific cause.
Check each in turn:
- Update settings:
yaml
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # one extra new pod at a time
maxUnavailable: 0 # never fewer than 3 ready podsWith maxUnavailable: 0, Kubernetes only removes an old pod after a new one is ready. A Recreate strategy, or a high maxUnavailable, causes gaps on purpose (question 351).
- Readiness. If readiness passes before the app has loaded its configuration, warmed its cache or connected to the database, traffic arrives too early and fails. Make readiness reflect "can serve" (question 404).
- Graceful shutdown. When a pod is removed, it keeps receiving traffic for a few seconds while load balancers update. The app must keep serving during that time, then finish its requests on
SIGTERM. A shortpreStopsleep and a long enoughterminationGracePeriodSecondscover the gap (question 356). - Replica count. With 1 replica, any restart is an outage. Run at least 2, better 3, spread across zones (question 396).
- Capacity. If the new surge pod is
Pendingbecause no node has room, the rollout waits. IfmaxUnavailableis above zero, Kubernetes may remove old pods first, reducing capacity during the gap. Check events forFailedScheduling(question 408). - Outside Kubernetes: a load balancer with slow health checks, or a gateway that caches pod addresses, can keep sending traffic to pods that are already gone.
Prove the fix: run a steady stream of requests during a rollout in staging, and count failures. The target is zero.
444. A database migration succeeds but the new application version fails; how would you restore service without corrupting data?
The short answer
Roll the application back to the previous version, but do not roll back the database blindly, because new data may already depend on the new schema. This only works safely if the migration was backward compatible, using the expand-and-contract approach: add new columns or tables without removing or renaming old ones, so the old version keeps working on the new schema. If the migration was not compatible, restore service with a quick forward fix or a carefully written reverse migration, and use point-in-time restore only as a last resort.
Why the order matters. The migration added a column and the new version started writing to it. Dropping that column now loses those writes. Rolling back the app, while keeping the new schema, loses nothing, as long as the old app ignores the column.
Expand and contract, the habit that makes this easy (question 287):
| Release | Database | App |
|---|---|---|
| 1, expand | Add the new column, nullable; keep the old one | Old version still works |
| 2 | Unchanged | New version writes both, reads new |
| 3, backfill | Copy old data into the new column | |
| 4, contract | Remove the old column, only after nothing uses it |
At every step, the previous app version still works, so rolling back the app is always safe.
When the migration was not backward compatible:
- Weigh a forward fix. If the app bug is small, a quick patched release may restore service faster and more safely than undoing the schema.
- A reverse migration written specifically for this case, run carefully, keeping any data written since the release.
- Point-in-time restore only if data is damaged, because it loses everything written since that moment. Restore to a new database, copy back what is needed, then switch (question 577).
Before the next one: run migrations as a separate, observable step before the app rollout (question 357), make them backward compatible by rule, and rehearse them against a copy of production data. The pipeline side of this same failure is in question 228.
445. A node fails during deployment; how would Kubernetes recover the workload, and what must the pipeline check before continuing?
The short answer
When a node stops responding, the control plane marks it NotReady, removes its pods from Service endpoints, and after a timeout, about five minutes by default, evicts them so their controllers create replacements on healthy nodes. A Deployment's rollout continues once enough new pods are ready elsewhere. The pipeline should not just check "rollout complete", but confirm the desired number of updated, ready replicas, no pods stuck Pending or Terminating, and healthy service signals, before it promotes or moves on.
What Kubernetes does, in order:
- The kubelet on the failed node stops sending heartbeats.
- After a grace period of under a minute, the node is marked
NotReady. - The node gets taints saying it is unreachable. Pods tolerate this for 300 seconds by default, then are evicted (question 394).
- Pods on that node become not ready, so Services stop sending them traffic.
- ReplicaSets create replacements on other nodes, if there is room. If not, the cluster autoscaler adds nodes (question 399).
- StatefulSet pods are more careful: a replacement is not started until the old pod is confirmed gone, to avoid two copies with the same identity and storage (question 340).
What the pipeline should check before continuing:
bash
kubectl -n shop rollout status deploy/checkout --timeout=10m
kubectl -n shop get deploy checkout \
-o jsonpath='{.status.updatedReplicas}/{.status.readyReplicas}/{.spec.replicas}'
kubectl -n shop get pods -l app=checkout --field-selector=status.phase!=Running- The rollout finished within the timeout.
- Updated, ready and desired counts match, such as
3/3/3. - No pods are left
Pending,Unknownor stuckTerminatingon the dead node. - Service signals are healthy: error rate and latency, not just pod counts (question 284).
Design so a node failure is boring: at least two or three replicas spread across nodes and zones, PodDisruptionBudgets, and spare capacity for one node's worth of pods (questions 396 and 400).