Appearance
When Pipelines Go Wrong
299. A pipeline is green, but production is running the previous version; how would you troubleshoot it?
The short answer
Compare what the pipeline thinks it deployed with what is actually running, one layer at a time. Did the deploy step target the right cluster and namespace? Did the desired state actually change, or did a reused tag such as latest leave nothing to update? Did the rollout finish, or is it stuck with the old pods still serving? With GitOps, did the tool sync the change? And is the app really reporting its own version, or is a cache serving old files?
Step 1: what is really running?
bash
kubectl get deploy checkout-api -n shop-prod -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' # (1)
kubectl get pods -n shop-prod -l app=checkout-api \
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[0].imageID}{"\n"}{end}' # (2)
kubectl rollout status deploy/checkout-api -n shop-prod --timeout=10s # (3)
curl -s https://shop.example.com/api/version # (4)- The image the Deployment asks for: its desired state.
- The image digest each pod is actually running. If a tag was reused, this can differ from what the pipeline built.
- Whether the rollout finished or is stuck.
- What the live app says about itself.
Step 2: match the symptom to the cause.
| What you find | Likely cause |
|---|---|
| Deployment still has the old image | The deploy step changed nothing: wrong cluster, wrong namespace, wrong kube context, or it updated a different file |
| Deployment has the new tag, pods have the old digest | A reused tag with imagePullPolicy: IfNotPresent: nodes used their cached old image (question 353) |
| Rollout stuck, old pods still serving | New pods failing readiness; the pipeline did not wait for the rollout (question 428) |
| GitOps tool shows "out of sync" or an error | Auto-sync off, a sync error, or the tool watches a different branch or path |
| GitOps tool reverted the change | Someone changed the cluster by hand and self-heal put the Git version back, or CI changed the cluster instead of Git |
Pods new, /version old | The front end is served from a CDN cache, or the image was built from an old checkout |
| Correct in one region, old in another | The deployment only reached some clusters |
Step 3: fix the cause, then close the gap that hid it. A green pipeline must mean "production now runs this digest". So the pipeline should, after deploying, wait for the rollout, then check that the running pods' digest and the live /version match what it built (question 230).
300. A deployment succeeds in staging but fails in production; what evidence would you gather before retrying?
The short answer
Collect the exact failure first: the error, the events, the logs, and the point in the process where it failed. Then compare production with staging in every way that matters: configuration and secrets, infrastructure limits and permissions, network access, data size and shape, scale, external dependencies and platform versions. Only retry once you know the cause and know that a retry is safe, because a retry in the middle of a migration or a partial rollout can make things worse.
1. The failure itself.
- The pipeline step that failed and its error message.
- Kubernetes events and pod logs, including the previous container's logs if it restarted (question 410).
- How far it got: did the migration run? Did some pods update?
2. What differs between staging and production:
| Area | Typical differences |
|---|---|
| Configuration | Environment variables, ConfigMaps, feature flags |
| Secrets | A production secret missing, expired or with a different format |
| Permissions | The production deploy identity lacks a role staging has |
| Network | Network policies, private endpoints, firewall rules, outbound allow-lists |
| Limits | Resource quotas, node sizes, rate limits |
| Data | A migration that is quick on staging's data and slow on production's |
| Scale | More replicas, more traffic during the rollout |
| Dependencies | Production credentials and limits of third-party services |
| Platform | A different Kubernetes version, missing custom resource definitions |
A side-by-side diff of the rendered configuration for both environments often finds the cause in minutes:
bash
diff <(kustomize build overlays/staging) <(kustomize build overlays/prod)3. Is a retry safe? Ask before pressing the button:
- If a migration half-ran, will running it again break or duplicate something?
- If some pods updated, is the mix of versions safe (question 304)?
- Is the failure something a retry can fix, such as a brief network error, or something it cannot, such as a missing permission?
A blind retry turns "we know where it failed" into "it failed twice, now in a different place".
301. A pipeline intermittently fails during dependency download; how would you distinguish an external outage from cache corruption or rate limiting?
The short answer
The error messages and the pattern of failures tell them apart. An outage shows server errors or timeouts from the package source, affects every build at the same time, and matches the provider's status page. Rate limiting shows "too many requests" errors, and follows build volume and a shared outgoing address. Cache corruption shows checksum or integrity errors, keeps failing on the runners that share that cache, and disappears when the cache is cleared. Network problems show TLS or DNS errors on some runners only.
| Cause | Typical error | Pattern | Clears up when |
|---|---|---|---|
| Upstream outage | 502, 503, timeouts | All builds, same time window | The provider recovers |
| Rate limiting | 429, toomanyrequests | Busy hours, shared IP, many parallel jobs | Volume drops, or you authenticate |
| Cache corruption | EINTEGRITY, checksum mismatch, broken archive | Same runners or cache key, repeatable | The cache is cleared |
| Network or proxy | TLS errors, DNS failures, connection resets | Some runners or networks only | Network fixed |
Gathering the evidence:
- Read the exact error lines from several failed runs, not just one.
- Plot failures against time. A tight cluster in one window points to an outage. Failures during the busiest hours point to rate limits.
- Group by runner and cache key. Failures tied to particular runners or one cache key point to corruption or local network problems.
- Check the provider's status page for the window.
- Retry with the cache disabled. If it passes every time without the cache, the cache is the problem.
The lasting fixes:
- A package proxy, such as Artifactory, Nexus or a cloud package service, that caches packages for all builds. Upstream outages and rate limits then barely matter (question 216).
- Authenticated pulls, for higher rate limits (question 477).
- Retries with backoff for network calls, only for errors that are temporary.
- Correct cache keys and integrity checks on install, so a corrupt cache is detected and rebuilt (question 223).
302. A release is blocked by a security scan, but the business needs the fix urgently; how would you proceed safely?
The short answer
Do not switch the scanner off. First find out whether the finding is new in this change or was already in production. If production already has it, shipping the urgent fix does not add risk, so a time-limited exception approved by security is reasonable. If the change introduces it, check whether it is exploitable and whether a quick fix exists. Then decide with security, record the exception with an owner, compensating controls and an expiry, ship, and follow up immediately.
Step 1: is the finding new? Compare with the image currently in production:
bash
trivy image --severity CRITICAL,HIGH --format json myacr.azurecr.io/shop/checkout-api:current > prod.json
trivy image --severity CRITICAL,HIGH --format json myacr.azurecr.io/shop/checkout-api:urgent-fix > new.jsonCompare the two lists of vulnerability IDs. Very often, the blocking finding is in the base image and already in production. The urgent fix does not make anything worse, and blocking it only keeps the original business problem alive.
Step 2: if it is new, how serious is it? Use the questions from question 137: severity, exploitation in the wild, whether the vulnerable code is reachable, and exposure.
Step 3: is there a quick fix? Often the fix is a base image or dependency bump that can go into the same release in minutes.
Step 4: decide together, and record it.
| Situation | Reasonable action |
|---|---|
| Finding already in production | Ship with a time-limited exception; fix the finding in the next release |
| New, low risk, no quick fix | Ship with an exception and a compensating control |
| New, serious, quick fix exists | Include the fix, then ship |
| New, serious, exploitable, no fix | Do not ship this way; find another route, such as a feature flag or a config change |
Every exception is recorded with a reason, an approver from security, compensating controls and an expiry date, using the same mechanism as question 279.
Step 5: follow up, the same day if possible. Fix the finding and remove the exception. Then look at why an urgent fix and a scanner failure met in the first place, for example a base image that had not been updated for months.
303. A production deployment takes 40 minutes and frequently times out; how would you find the bottleneck?
The short answer
Break the 40 minutes into its pieces with timestamps, then fix the largest piece. Typical pieces are waiting in queues and for approvals, and pulling large images onto nodes. Waiting for new nodes when the cluster is full adds more. So do slow application start-up and readiness, and a rollout configured to replace one pod at a time. Disruption budgets and long shutdown grace periods slow the removal of old pods. Migrations, canary pauses and deployment hooks add the rest. Kubernetes events and the pipeline's step timings show where the time goes.
Step 1: build a timeline. Collect timestamps from the pipeline steps and from Kubernetes:
bash
kubectl get events -n shop-prod --sort-by=.lastTimestamp | grep checkout-api
kubectl rollout history deploy/checkout-api -n shop-prodEvents show each pod being scheduled, pulling its image and how long it took, starting, and becoming ready.
Step 2: look for the usual suspects.
| Piece | Sign | Fix |
|---|---|---|
| Waiting for a runner or an approval | Long gap before the deploy job starts | More runners; faster approvals (question 290) |
| Image pulls | "Pulling image" takes minutes | Smaller images, a registry in the same region (question 158) |
| No capacity | Pods Pending until new nodes join | Headroom, or faster node provisioning (question 399) |
| Slow start-up | Long time from "started" to "ready" | Faster start-up; a startup probe instead of a long initial delay |
| One pod at a time | maxSurge: 1, maxUnavailable: 0 with many replicas | Higher maxSurge (question 351) |
| Slow removal | Long grace periods, disruption budgets waiting | Shorter shutdown; sensible budgets |
| Migrations | Migration job runs for many minutes | Online migrations, separated from the deployment (question 227) |
| Canary pauses | Long pauses by design | Shorter steps for low-risk changes |
| Hooks | Helm hooks waiting on jobs | Faster jobs, or move them out |
A worked example: 30 replicas, maxSurge: 1, maxUnavailable: 0, and each new pod takes 70 seconds to become ready. The rollout replaces pods one at a time: 30 × 70 seconds is 35 minutes. Setting maxSurge: 25% lets about eight new pods start at once, and the rollout takes roughly five minutes.
Step 3: align the timeouts. Set both the Deployment's progressDeadlineSeconds and the pipeline's own timeout from the expected duration plus a margin. Then a real problem is reported quickly, and a normal rollout is not cut off.
304. A failed deployment leaves some instances on the new version and others on the old version; how would you restore consistency?
The short answer
Decide which version should be running, then make the platform converge on it from its source of truth, instead of fixing instances by hand. If the new version is broken, roll back to the previous known-good version. If it is sound and the failure was elsewhere, finish the rollout. Check whether the mix of versions is harming users in the meantime. Then confirm every instance reports the same version, and find out why the rollout stopped halfway.
Step 1: is the mix itself harmful right now? If the two versions are compatible, users may be fine, and there is time to think. If not, for example they read and write data differently, act fast.
Step 2: choose the target version.
- The new version is the problem: roll back to the last known-good version.
- The new version is fine, and the rollout stopped for another reason, such as a node failure or a timeout: finish it.
Step 3: let the platform converge.
bash
kubectl rollout undo deploy/checkout-api -n shop-prod # back to the previous ReplicaSet
kubectl rollout status deploy/checkout-api -n shop-prod --timeout=10mWith GitOps, change the configuration repository back to the known-good digest, or forward to the one you want, and let the tool sync. That keeps Git as the record of what should be running.
Step 4: verify consistency.
bash
kubectl get pods -n shop-prod -l app=checkout-api \
-o jsonpath='{range .items[*]}{.status.containerStatuses[0].imageID}{"\n"}{end}' | sort | uniq -cThe output should show one digest, with a count equal to the number of replicas.
Step 5: find out why it stopped halfway, and make the deployment safer: health-gated rollouts with automatic rollback, a progress deadline, and a pipeline that waits for and verifies the result (question 230).
Outside Kubernetes, for example on virtual machines, the principle is the same: redeploy the chosen version to every instance from the pipeline, never by hand on individual machines.
305. How would you design a CI/CD platform for dozens of teams that need standardized security but different deployment workflows?
The short answer
Separate what must be the same from what may differ. Build a paved road: versioned, reusable pipeline templates that contain the required security steps, such as scanning, signing and provenance, with clear extension points for each team's own stages. Enforce the non-negotiable controls at the platform and at the deployment target, through branch rules, required workflows and admission policies, rather than trusting each pipeline. Give teams self-service onboarding, isolated runners and identities, shared monitoring, and good documentation.
1. The paved road. Reusable workflows or templates for each common kind of service, with the security steps built in, versioned so teams upgrade on their own schedule (question 259).
2. Extension points. Inputs and hooks where teams add their own stages, choose a rollout style, or add checks.
3. Enforcement underneath, so security does not depend on every pipeline being right:
- Branch rules and required workflows: platforms such as GitHub can require certain workflows, for example the security scan, to run on every pull request in chosen repositories, whatever the repository's own files say.
- Identity: OIDC trust per repository and environment, so a pipeline can only reach its own cloud resources (question 265).
- Admission control: clusters accept only images signed by approved workflows, with provenance and recent scan results (question 147).
4. Self-service. A new service gets its repository, pipeline, registry repository, namespaces and identities from one request or one template, in minutes.
5. Isolation: runner groups and secrets per team (question 265).
6. Visibility: shared dashboards for pipeline health and DORA metrics per team (questions 266 and 294).
7. People: documentation, examples, office hours, and a way for teams to contribute improvements to the templates.
306. How would you recover a CI/CD platform after loss of its controller, configuration, or artifact repository?
The short answer
Recovery is only possible if you prepared for it. Keep the platform's configuration as code, pipelines in their repositories, secrets in a backed-up vault, and artifacts replicated or backed up. Then recover in order: identities and secrets first, then the controller from code and backups, then the runners, then the artifact store, then prove it works with a real pipeline run. If artifacts are truly lost, production keeps running on cached images while you rebuild from source and carefully re-sign and re-promote.
Preparation, which is most of the answer:
| Component | Make it recoverable by |
|---|---|
| Controller configuration | Configuration as code, such as JCasC for Jenkins (question 240) |
| Pipelines | Pipeline files in each repository |
| Controller state | Regular, tested backups, including encryption keys |
| Secrets | A vault with its own backups; documented rotation |
| Runners | Built from infrastructure as code and pinned images |
| Artifacts and images | Replication to another region, plus backups |
| Runbook | Written steps, tried in a drill (question 307) |
Recovery order, and why:
- Identity and secrets. Nothing else can authenticate without them.
- The controller, rebuilt from code and restored from backup.
- Runners, recreated from their templates.
- The artifact store, restored or failed over to a replica.
- Proof: run a real pipeline end to end, from a commit to a deployment in a test environment.
If the artifact repository is truly lost:
- Production keeps running. Nodes have the images they already pulled, so existing pods continue. But new pods cannot start on new nodes, so avoid scaling events and node maintenance until recovery (question 446).
- Rebuild from source. With reproducible builds, rebuilt images can even match the old digests (question 85). Without them, the new images are new artifacts: they must be scanned, signed and promoted through the normal gates again, not just pushed.
Set targets in advance: how long the platform may be down, its RTO, recovery time objective, and how much data, such as build history, may be lost, its RPO, recovery point objective (question 584).
307. How would you design disaster recovery and backup tests for a release platform?
The short answer
Start from targets: how quickly the platform must be able to deploy again after a disaster, and how much may be lost. List every part the release process depends on, including outside services such as the Git host. Then test recovery for real, on a schedule: tabletop walkthroughs, restore drills into an isolated environment, and planned failover exercises. Measure the time until the first successful deployment after the simulated disaster, record every gap found, and fix the gaps before the next test.
1. Set targets. For example: "after the loss of the CI controller, we can deploy an urgent fix to production within 2 hours", and "we lose at most 24 hours of build history".
2. List dependencies. A release needs more than the CI server:
- Git hosting, which may be an outside service.
- The CI controller and its runners.
- The artifact repository and the container registry.
- The secrets store.
- The GitOps controller and the configuration repository.
- Identity: OIDC trust, single sign-on.
- DNS and networks.
For each, ask: what if it disappears? For outside services, the answer may be "we keep a mirror of critical repositories and a documented emergency deployment path".
3. Test at increasing levels:
| Test | What happens | How often |
|---|---|---|
| Tabletop | People walk through the runbook, step by step, on paper | Quarterly |
| Restore drill | Restore backups into an isolated environment and check them | Quarterly |
| Failover exercise | Switch to the replica registry or a standby controller for real | Twice a year |
| Game day | A realistic simulated disaster, with the on-call team responding | Yearly |
4. Measure the right thing. "Backup restored" is not the goal. "Time until a real change deploys through the restored platform" is.
5. Verify restored data. Check that restored artifacts' checksums match the originals and that signatures still verify, so you know the backups are both complete and untampered.
6. Fix and repeat. Every test produces a list of gaps: an expired credential in the runbook, a missing backup of an encryption key, a step only one person knew. Fix them, and test again.
308. How would you stop a compromised pipeline from deploying arbitrary code to production?
The short answer
Design so that a pipeline alone cannot deploy just anything. Production accepts only images that are signed, with provenance showing they were built by the protected release workflow from reviewed code, and the cluster itself checks this. Deployment happens through a reviewed configuration repository, with GitOps pulling from it, and approvals on the production environment. Build and deploy identities are separate and narrow. If a compromise is suspected: revoke credentials, freeze deployments, compare everything running with release records, and roll back to known-good versions.
Prevention, so a compromise cannot become a deployment:
- Verification at the destination. The cluster's admission policy accepts only images signed by the release workflow on main, with matching provenance (question 147). An attacker who controls some pipeline, but not the protected main branch and its review rules, cannot produce that.
- Pull-based deployment. CI writes a digest into a configuration repository through a reviewed pull request, and a GitOps tool inside the cluster pulls it. CI holds no cluster credentials at all (question 271).
- Approvals on the production environment, which pipeline files cannot remove (question 257).
- Separate, narrow identities. The build identity can push images to one repository, nothing else. The deploy identity can update one namespace, nothing else. OIDC trust is bound to exact repositories and environments (question 269).
- Protected pipeline definitions. Changes to workflows and templates need code-owner review (question 261).
Detection, so a compromise is noticed quickly:
- Alerts on deployments that did not come from the normal path, and on running digests that no release record explains.
- Alerts on unusual credential use, such as a deploy role used at 3 a.m. from an unexpected workflow.
Response, if it happens:
- Contain: disable the compromised pipeline, revoke and rotate its credentials, and remove OIDC trust for the affected repositories.
- Freeze production deployments.
- Investigate: list everything deployed during the window, and compare every running digest with signed release records (question 454).
- Recover: redeploy known-good versions from trusted artifacts, rotate any secret the pipeline could have read, and rebuild runners from clean images.
- Learn: an incident review, and closing the gap that allowed it.