Skip to content

Interview PrepScenarios

Incidents and Judgment ​

446. The image registry becomes unavailable during a production rollout; what happens to existing Pods and new Pods, and how would you reduce the risk? ​

The short answer

Running pods keep running, because their images are already on their nodes. New pods start only if their node already has the exact image and the pull policy allows using it, so pods placed on fresh nodes, or using imagePullPolicy: Always, fail with ImagePullBackOff. A rolling update with maxUnavailable: 0 simply stalls, keeping the old pods serving. Reduce the risk with a highly available or geo-replicated registry, a pull-through cache, deploying by digest with IfNotPresent, and alerts on pull failures.

registryunavailablenode 1: running podsimage already on disk; keeps servingnode 2: new pod, image cachedIfNotPresent and same digest: starts without the registrynode 3: new node, no imagemust pull: ImagePullBackOff until the registry returnspull fails
The registry matters only at pull time. Running pods do not need it. New pods need it only when their node does not already hold the image.

What happens, case by case:

SituationResult
Pods already runningUnaffected
Pod restarts on the same node, image cached, IfNotPresentStarts normally
imagePullPolicy: AlwaysFails: it always asks the registry, even if the image is cached (question 353)
New node from autoscaling, or a replacement for a failed nodeFails to pull
Rolling update with maxUnavailable: 0New pods stall; old pods keep serving; the rollout waits
Rolling update with high maxUnavailableOld pods removed, new ones cannot start: capacity drops

The hidden danger is not the rollout, which stalls safely. It is a node failure or a traffic spike during the outage: new pods cannot start anywhere new, so the cluster cannot heal or scale.

Reducing the risk:

  1. A highly available registry, such as a zone-redundant registry tier, or geo-replication to each region you run in (question 135).
  2. A pull-through cache or mirror inside your network, so nodes pull from nearby and keep working when the upstream is down.
  3. Deploy by digest with IfNotPresent. A digest never changes, so a cached copy is always correct, and no registry call is needed.
  4. No dependency on public registries at runtime: copy base and third-party images into your own registry (question 477).
  5. Retention rules that never delete deployed images, so a rollback can always pull.
  6. Alert on pull errors, and pause automatic rollouts when the registry is unhealthy.

447. How would you deploy a service using canary traffic while monitoring errors and latency before full rollout? ​

The short answer

Run the new version next to the stable one and send it a small share of traffic, such as 5%. A rollout controller, such as Argo Rollouts or Flagger, compares the canary's error rate and latency with set thresholds, using queries against your metrics, for a few minutes at each step. If they pass, it raises the share step by step to 100%. If any check fails, it moves all traffic back to the stable version and stops, with no human needed.

An Argo Rollouts canary for checkout:

yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
  namespace: shop
spec:
  replicas: 10
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: app
          image: ghcr.io/shop/checkout@sha256:9f2c...
  strategy:
    canary:
      canaryMetadata:
        labels: { track: canary }                      # (1)
      analysis:
        templates:
          - templateName: checkout-health              # (2)
        startingStep: 1
      steps:
        - setWeight: 5                                 # (3)
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }                     # (4)
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-health
  namespace: shop
spec:
  metrics:
    - name: error-rate
      interval: 1m
      failureLimit: 2                                  # (5)
      successCondition: result[0] < 0.005
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="checkout",track="canary",status=~"5.."}[2m]))
            / sum(rate(http_requests_total{app="checkout",track="canary"}[2m]))
    - name: p99-latency
      interval: 1m
      failureLimit: 2
      successCondition: result[0] < 0.8
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            histogram_quantile(0.99, sum by (le) (
              rate(http_request_duration_seconds_bucket{app="checkout",track="canary"}[2m])))
  1. Label canary pods track: canary, so metrics can be split between canary and stable.
  2. Background analysis runs the checks continuously from step 1 onwards.
  3. 5% first. Without a traffic router, Argo approximates weights with pod counts; with a gateway or service mesh, the split is exact.
  4. Each step waits while analysis keeps checking. After the last step, the rollout goes to 100%.
  5. Two failed checks abort the rollout: traffic returns to stable, and the canary is scaled down.

The two queries are the error-rate and p99 formulas from question 626, filtered to canary pods.

What makes canaries trustworthy:

  • Enough traffic in the canary for the numbers to mean something. At 5% of 10 requests per second, ten minutes is only 300 requests.
  • Compare with stable, not only with fixed numbers, so normal daily patterns do not cause false alarms (question 283).
  • Business signals too, such as completed checkouts, not only technical ones (question 293).
  • Sticky sessions where needed, so one user does not flip between versions on every request.

448. How would you design a safe rollback when application configuration and container image versions change together? ​

The short answer

Version the configuration together with the image, so one release is one unit: image digest plus configuration, stored in Git and deployed together. Give each configuration version its own name, for example a ConfigMap with a content hash in its name. Then a new version creates new pods that point at the new configuration, while the old pods keep the old one. Rolling back then means returning to the previous unit, through a Helm rollback or a Git revert, which restores both at once.

The problem with in-place configuration. The release changes the image to version 2 and edits the checkout-config ConfigMap. Version 2 fails, and you roll back the Deployment to version 1. But the ConfigMap is still the new one, which version 1 may not understand. Now the rollback fails too.

The fix: name configuration by its content. Kustomize's configMapGenerator adds a hash of the content to the name:

yaml
configMapGenerator:
  - name: checkout-config
    literals:
      - PAYMENT_TIMEOUT_MS=3000
      - CURRENCY_DEFAULT=EUR

This produces a ConfigMap such as checkout-config-7b9t2m6k4f, and rewrites the Deployment to point at that exact name. Changing a value produces a new name, which changes the pod template, which starts a normal rolling update. The old ConfigMap still exists, so old pods, and a rollback, keep using the old values (question 380).

The release unit and its rollback:

ToolOne release isRollback
HelmA revision: rendered manifests including config and imagehelm rollback checkout 41 (question 505)
GitOpsA commit in the environment repositoryRevert the commit; Argo CD or Flux sync it (question 512)
Kustomize with hashed ConfigMapsThe rendered output of one commitApply the previous commit

Also make configuration backward compatible, like database changes: new settings get defaults, so the old version ignores them and the new version works without them. Then image and configuration can even roll back separately when needed. Secrets follow the same idea with versioned secret names or versions in the secret store.

449. How would you investigate a release where the pipeline says the new commit was deployed but the running Pods use an older image digest? ​

The short answer

Compare what the pipeline changed with what the cluster is running, step by step. Check that it changed the right cluster, namespace and Deployment. Check whether the Deployment's spec has the new image but its rollout is stuck, so old pods are still serving. Check whether something reverted the change: a GitOps controller syncing an older Git state, or another pipeline or Helm release managing the same object. And check tag problems: a reused tag with IfNotPresent can leave nodes running an old image under a new-looking name.

Step by step:

  1. What does the Deployment ask for?
bash
kubectl -n shop get deploy checkout -o jsonpath='{.spec.template.spec.containers[0].image}'

If it is the old image, the change never landed, or was reverted. If it is the new one, the rollout is the problem.

  1. Right target? The pipeline's kube context, cluster, namespace and resource name. Pipelines that deploy to the wrong cluster report success happily (question 299).
  2. Is the rollout stuck?
bash
kubectl -n shop rollout status deploy/checkout
kubectl -n shop get rs -l app=checkout

A new ReplicaSet with zero ready pods means new pods fail to start, and old pods keep serving (questions 434 and 433).

  1. Was it reverted? In GitOps, Argo CD or Flux applies what Git says. A pipeline that ran kubectl set image directly gets undone at the next sync, because Git still has the old digest. Check the application's sync history (question 508).
  2. Two owners? A Helm release and a kubectl apply pipeline both managing checkout fight each other. The managedFields in the object show who last changed which field.
  3. Mutable tags. If the image is checkout:latest or a reused version tag with imagePullPolicy: IfNotPresent, nodes that cached an older image under that tag keep using it (question 353). The fix is deploying by digest.
  4. Admission webhooks can rewrite images, for example to a mirror registry. Compare the spec with the pod's imageID.

Prevent it: deploy by digest, have one owner per object, use GitOps as the only path, and make the pipeline's final check compare the digest running in pods with the digest it built (question 432).

450. How would you design a release process that supports auditability, least privilege, artifact signing, and a tested rollback path? ​

The short answer

Make Git the record of every change and the pipeline the only way to production. Each release is a reviewed pull request, built once, signed, with an SBOM and provenance. Production changes run under short-lived, narrowly scoped identities, with human approval recorded on the environment. The cluster only accepts signed images from the trusted pipeline. Rollback is the same path backwards, a revert or a previous signed digest, and it is practised regularly so it is known to work.

NeedHow
Who changed what, and whyPull requests with reviews; commit messages linking tickets (question 196)
What was deployed, when, by whomDeployment records per environment: digest, commit, approver, time
Least privilegeOIDC identities per environment; people read-only in production (question 271)
Only trusted artifactsSigning and provenance in the pipeline; verification in the cluster (question 275)
ApprovalsEnvironment protection rules; approvals stored by the platform (question 257)
RollbackPrevious signed digest, or a Git revert; same pipeline, same checks
Proof rollback worksRollback drills in staging and occasionally in production
Tamper-evident historyAudit logs from Git, CI, cloud and cluster sent to central storage (question 555)

A release, end to end:

  1. A developer opens a pull request; checks and a reviewer approve it.
  2. Merge builds one image, signs it, and attaches an SBOM and provenance.
  3. The pipeline writes the digest to the staging folder of the GitOps repository; Argo CD deploys it.
  4. After staging checks pass, a promotion pull request copies the digest to production, and a second person approves.
  5. The production cluster verifies the signature at admission, and the canary analysis decides the rollout.
  6. Every step leaves a record that links commit, digest, approvers and time.

Rollback uses the same path. Reverting the production commit deploys the previous digest, which is already signed and verified. No special "emergency" route skips the checks, so emergencies do not become security holes (question 298).

Keep it practised. A rollback that has not been run in months may fail on something nobody noticed, such as a deleted old image or an incompatible database change (question 444).

451. A pipeline passes all tests, but a Kubernetes readiness probe repeatedly fails after deployment; how do you determine whether the probe or the application is wrong? ​

The short answer

Ask the app directly, from inside the pod, exactly what the probe asks: the same port, path, headers and timeout. If the app answers quickly and correctly there, the probe definition is wrong, such as the wrong port, the wrong path, or a timeout that is too short. If the app does not answer, or answers with an error, the app is wrong: read its logs for the reason, such as a dependency it cannot reach or start-up work not finished. The pod's events show the probe's exact failure message.

Step by step:

  1. Read the failure message:
bash
kubectl -n shop describe pod checkout-7d9f-x2 | grep -A3 -i readiness

Messages such as connection refused, context deadline exceeded or HTTP probe failed with statuscode: 503 say different things.

  1. Compare the probe with the app:
yaml
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3

Is the app really listening on 8080, and does /health/ready exist in this version?

  1. Ask the app the same question from inside the pod, or from a debug container if the image has no tools (question 414):
bash
kubectl -n shop exec checkout-7d9f-x2 -- wget -qO- -T 1 http://127.0.0.1:8080/health/ready
ResultPoints to
Quick 200 here, probe still failsThe probe: wrong port name, wrong path, or the app only listens on 127.0.0.1 while the kubelet calls the pod IP
connection refusedThe app is not listening on that port: wrong port, or it has not started yet
Slow answer, over 1 secondTimeout too short, or the endpoint does heavy work
503 with a reasonThe app says it is not ready: read its logs for why
  1. Read the app's logs for the reason it reports itself not ready, such as "cannot connect to Redis", which may be a network policy, a missing Secret, or a new dependency the cluster lacks.

Why the pipeline passed. Its checks ran without the cluster's real network, secrets, policies and dependencies. Readiness is exactly where those differences show up, which is why it is valuable (question 436).

452. How would you diagnose a service that fails only under load because of CPU throttling, memory limits, or insufficient replicas? ​

The short answer

Reproduce the load in staging or watch production during a peak, and look at four things together: CPU throttling, memory against limits, replica count against the autoscaler's maximum, and the service's latency and errors. Throttling shows as latency rising while CPU use sits flat at the limit. Memory problems show as OOMKilled restarts. Too few replicas shows as the autoscaler at its maximum, or reacting too slowly, while each pod is saturated. Fix the specific cause, then raise the load again to confirm.

What each cause looks like:

CauseSignalsFix
CPU throttlingThrottled-period ratio high; CPU flat at the limit; p99 latency upRaise or remove the CPU limit, or right-size requests (question 392)
Memory limitOOMKilled, restarts under load, memory climbing to the limitRaise the limit after checking it is not a leak (question 411)
Not enough replicasHPA at max replicas, or scaling too slowly; every pod busyRaise max, scale on a better metric, start scaling earlier (question 398)
No room for new podsNew pods Pending during the spikeCluster autoscaler limits, node pool size (question 399)
Downstream limitDatabase connections exhausted; dependency latency upPool sizes, caching, protecting the dependency (question 582)

The throttling query:

text
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="shop",container="app"}[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="shop",container="app"}[5m]))

It gives, per pod, the share of CPU time slices in which the container wanted more CPU than its limit allowed. Values above about 0.25 usually mean latency is suffering.

Reproducing it. Use a load generator, such as k6, against staging with the same limits and replica settings as production, and raise the rate step by step while watching the dashboard. The first signal to bend tells you the bottleneck.

Common traps:

  • HPA on CPU with throttled pods. A throttled pod cannot show more than its limit, so CPU utilisation looks capped and scaling may lag. Scaling on request rate or latency can work better (question 397).
  • Slow start-up. New pods that take a minute to become ready arrive too late for a sudden spike. Keep more headroom, or scale on a schedule before known peaks.

453. How would you design separate development, test, staging, and production environments while minimizing configuration drift? ​

The short answer

Build every environment from the same code: the same infrastructure modules and the same Kubernetes manifests, with a small, explicit set of differences per environment, such as size and addresses. Promote the same artifacts, the same image digests, from one environment to the next. Make Git and the pipeline the only way to change any environment, detect drift automatically, and keep differences visible and few, so staging behaves like production in every way that matters.

One base, small differences:

text
deploy/
  base/                   the manifests for every environment
  overlays/
    dev/                  1 replica, small requests, debug logging
    test/                 2 replicas
    staging/              3 replicas, production-like data shape
    prod/                 6 replicas across 3 zones, strict policies

Each overlay only patches what truly differs. A reviewer can read the full list of differences in a few files (question 506).

The same idea for infrastructure: one Terraform module per component, called from each environment's folder with its own settings and its own state (question 602).

Rules that keep environments close:

RulePrevents
Promote digests, never rebuild per environment"It worked in staging" with a different build (question 134)
Same versions of Kubernetes, add-ons and managed servicesSurprises from version differences
Same policies, in audit mode in dev if neededA deployment that only fails in production
No manual changes; read-only access for peopleHand edits that nobody records (question 600)
Scheduled drift detectionSlow, silent divergence
Differences listed in one placeHidden, forgotten differences

Where differences are fine: scale, cost-saving choices such as smaller databases, data, and secrets. Where they are dangerous: networking rules, security policies, versions, and the deployment process itself.

Short-lived environments, created per pull request from the same code and deleted after, help teams check changes in isolation without keeping many long-lived copies to drift.

454. How would you handle a compromised deployment credential and verify that no unauthorized workloads or images were deployed? ​

The short answer

Revoke the credential immediately and replace it, then investigate. Use audit logs from the cloud, the Kubernetes API, the registry and the CI system to list everything that credential did since it might have leaked. Compare what is running in every cluster with what your release records say should be running, digest by digest, and look for anything new: workloads, CronJobs, DaemonSets, RBAC bindings, webhooks and images. Remove what should not exist, and close the gap that let the credential leak.

In order:

  1. Revoke and rotate. Disable the credential, such as a service account key or kubeconfig token. Rotate anything it could read, because those secrets may be stolen too (question 267).
  2. Scope the window. When could it have leaked? Use the earliest possible time, not the time you noticed.
  3. Read the audit logs for that identity: Kubernetes API audit logs, cloud activity logs, registry push logs, CI logs (question 555).
  4. Compare running with expected:
bash
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | sort -u

Every image should match a digest from your signed release records. Anything else is suspicious.

  1. Look for persistence, the things an attacker adds to keep access:
Look forWhy
New ClusterRoleBindings or RoleBindingsGranting themselves lasting rights
New service accounts and tokensA credential that survives your rotation
DaemonSets, CronJobsCode that runs on every node, or keeps coming back
Mutating admission webhooksSilently changing future deployments
Images pushed outside the pipelinePoisoned tags waiting to be deployed
Cloud resources: new users, keys, machinesAccess and crypto mining beyond the cluster (question 558)
  1. Remove and redeploy from known-good sources: Git and signed digests.
  2. Fix the cause and add controls: short-lived federated credentials instead of keys, admission policies that only accept signed images, and alerts on unexpected RBAC changes (questions 308 and 500).

455. What would you include in a post-incident review after a failed release, and how would you turn the findings into automated prevention? ​

The short answer

Include a factual timeline and the impact on users and the business. Say how the problem was detected and how long each stage took. List the contributing causes across code, process and tooling, and what helped and what hurt. End with specific actions, each with an owner and a date. Keep it blameless. Then turn each lesson into something automatic where possible: a pipeline check, an admission policy, an alert, a canary analysis rule or a safer default, so the protection does not depend on people remembering.

What the review contains:

  1. Summary: what happened, in three sentences.
  2. Impact: users affected, duration, failed orders, error budget used (question 631).
  3. Timeline: release time, first errors, alert, acknowledgement, diagnosis, rollback, recovery (question 638).
  4. Contributing causes: usually several, such as "the config key was renamed", "staging used a default value", and "the canary did not check checkout success".
  5. What went well: such as "rollback took 2 minutes".
  6. What made it worse: such as "the alert went to an old channel".
  7. Actions, each with an owner and a due date.

From finding to automation:

FindingAutomated prevention
A renamed config key broke start-upStart-up validation of required settings; the pod fails readiness, not users
Staging had a default production lackedSame overlays, with a check that lists differences
The canary missed broken checkoutsAdd checkout success rate to the canary analysis (question 447)
An unsigned image reached productionAdmission policy rejecting unsigned images (question 500)
Detection took 25 minutesSLO burn-rate alert on the checkout journey (question 631)
Rollback was slow and manualAutomated rollback on failed rollout in the pipeline (question 431)

Follow through. Track actions like any other work, review them in a regular meeting, and close the review only when they are done. A review whose actions never happen is only a story.

456. How would you prioritize the first five checks when production is down immediately after a release? ​

The short answer

First, confirm the scope from the user-facing signals: what is broken and for whom. Second, since it started right after a release, roll back or turn off the change straight away, without waiting for the root cause. Third, check whether the rollback restored service. Fourth, if it did not, look at what else changed at the same time, such as configuration, infrastructure or a dependency, and the health of shared parts. Fifth, communicate and gather evidence for the review.

① scopewhat fails,for whom② roll backor flag off,now③ recovered?same signalsas step ①④ look widerconfig, infra,dependencies⑤ tell, recordupdates andevidencerestore first, explain later; communication runs alongside from the start
Restore first, explain later. Confirm what is broken, undo the most likely cause, check it worked, and only then widen the search. Updates to people run alongside every step.

Each step, in a little more detail:

  1. Scope. The SLO dashboard and error rate by route: is it everything, one journey, one region? This takes a minute and guides everything else (question 629).
  2. Roll back. A release just went out and production broke: the odds strongly favour the release. Roll back, or turn off its feature flag, before debugging (question 427). Do not wait for proof.
  3. Recovered? Watch the same signals for a few minutes. If they return to normal, service is restored, and the root cause can wait for working hours.
  4. Look wider, if not. Something else changed too, or the release changed something a rollback does not undo, such as a database migration or a configuration change outside the image. Check recent changes of every kind, shared dependencies, and the platform itself (question 423).
  5. Communicate and record. Declare the incident, post updates on a schedule, and keep a timeline of every action. Evidence gathered now makes the review honest later (question 637).

What not to do first: read code, debug in production, or deploy a "quick fix" forward before trying the rollback. A forward fix under pressure is how one incident becomes two.

457. How would you explain the trade-offs of faster releases, stricter quality gates, and operational risk to a product manager? ​

The short answer

Explain that speed and safety are not opposites: years of DORA research show the best teams release more often and fail less, because small, frequent changes are easier to check, easier to roll back and less risky each. Strict gates help when they are fast and automatic, and hurt when they are slow and manual, because they push teams to batch changes. Use shared numbers, such as the SLO and error budget, to decide together: while the budget is healthy, ship faster; when it is spent, slow down and invest in reliability.

Start from what they care about: features reaching customers, and customers not being hurt. Then explain with numbers, not opinions.

Three points that usually land:

  1. Small batches are safer. A release with one change that breaks is fixed in minutes, because you know what changed. A release with fifty changes that breaks takes hours to untangle.
  2. Gates have a cost. A manual approval that takes two days means work piles up, and the next release is bigger and riskier. An automatic check that takes two minutes is nearly free (question 290).
  3. Risk can be priced. "An hour of checkout downtime costs about £40,000" turns reliability into a business decision the product manager can weigh against features.

A shared rule, so it is not a negotiation each time:

Error budgetWhat it means for delivery
HealthyShip as fast as the pipeline allows
Getting lowSmaller changes, more canary time
SpentFeature releases pause; fixes and reliability work first (question 631)

Show the measures together, both speed and stability: deployment frequency and lead time next to change failure rate and recovery time. If speed improves and failures do not rise, the approach is working (question 294).

Offer options, not refusals. Instead of "we cannot release on Friday", say "we can release on Friday behind a feature flag, turned on for 5% of users, with automatic rollback".

458. How would you keep the deployment system simple enough to operate while still meeting security, availability, and audit requirements? ​

The short answer

Offer one well-built path that every team uses, instead of many custom ones, and build it from managed services and widely used tools rather than home-made ones. Put security, audit and availability inside that path as defaults, such as signing, OIDC, approvals and canaries, so teams get them without extra work. Remove tools and steps that do not earn their keep, document the path, and measure it. Complexity should be added only when a real requirement cannot be met otherwise.

The idea of a paved road. A platform team builds one supported way to go from code to production: a template repository, a shared pipeline, a standard deployment chart, GitOps and default dashboards. Teams can step off it, but then they own the extra work. Most teams stay on it because it is the easiest way (question 305).

Simplicity choices that still meet strict requirements:

RequirementSimple way to meet it
AuditGit as the record; pipeline and platform logs kept centrally
Least privilegeOIDC federation per environment instead of managing keys
Supply-chain securitySigning and SBOMs built into the shared pipeline once
AvailabilityManaged Kubernetes or container services; rolling updates and PDBs by default
ApprovalsThe CI platform's environment protection, not a custom ticket bot
RollbackGit revert or Helm rollback, the same for every team

Signs the system is getting too complex:

  • Nobody can explain the full path from commit to production on a whiteboard.
  • Several tools do the same job, such as two deployment tools in one cluster.
  • Incidents are often caused by the delivery system itself.
  • New team members take weeks to ship their first change.

Rules to keep it small: every new tool must replace something or meet a written requirement; prefer configuration of an existing tool over a new one; and delete unused pipelines, jobs and plugins regularly.

459. If you inherited a fragile pipeline and an unfamiliar Kubernetes cluster, what would you inspect first and why? ​

The short answer

Learn before changing anything. First make sure you can recover: that access works, backups and the source of truth exist, and you know how to roll back. Then map what exists: which services run, how they get deployed, which versions are in use and whether they are still supported, and what monitoring and alerts exist. Then look at recent pain: incidents, failed runs and manual workarounds. Fix the biggest risks first, such as missing backups or unsupported versions, and make improvements in small, reversible steps.

Week one, in order:

  1. Access and recovery. Do you have working, personal access with the right rights? Are there backups of anything stateful? Is there a way to recreate the cluster and the pipelines from code? If not, that is the first risk.
  2. Inventory. What runs in the cluster: namespaces, workloads, add-ons, controllers and their versions.
bash
kubectl version
kubectl get nodes -o wide
kubectl get ns
kubectl get deploy,sts,ds,cronjob -A
helm list -A
  1. Support status. Kubernetes version and its support end date, node images, and add-ons such as the ingress controller, which may be retired (question 491). Unsupported versions block safe upgrades later.
  2. How things get deployed. Every pipeline that touches the cluster, which identity it uses, and whether there are manual steps or hand-applied resources that exist nowhere in Git (question 600).
  3. Observability. Which dashboards and alerts exist, who receives them, and whether they fire for real problems.
  4. Recent pain. Incident history, failed pipeline runs and their causes, and the workarounds people use. Ask the team where it hurts.
  5. Security basics. Who has cluster-admin, stored long-lived credentials, public endpoints, and images from unknown sources (question 415).

Then improve gradually:

  • Write down what you learned, as a map and a list of risks.
  • Fix the risks that could cause permanent loss first: backups, single points of failure, credentials.
  • Change one thing at a time, with a way back, and measure the effect.
  • Avoid rewriting everything at once. The current system, however fragile, works in ways you may not yet understand.

460. What evidence would convince you that the deployment process is reliable rather than merely successful on its last run? ​

The short answer

Trends over many releases, not one green run. Look at the DORA measures over months: frequent deployments, short lead time, a low change failure rate and fast recovery. Look at the share of pipeline runs that fail for reasons unrelated to the code, such as flaky steps or runner problems. Check that rollbacks and failovers are practised and work, that every service uses the same path, that builds are reproducible and traceable, and that problems are caught by automation before users notice them.

Evidence, and what it shows:

EvidenceShows
DORA measures over months (question 294)Speed and stability together, not one good week
Change failure rate, trendHow often releases hurt users
Time to restore after bad releasesWhether failures are cheap
Pipeline failures not caused by codeFlakiness and platform health (question 266)
Rollback drills: frequency and durationWhether rollback is real or only on paper
Share of services on the standard pathWhether the process is used, or bypassed
Every production digest traceable to a reviewed commitControl and audit (question 440)
Incidents found by alerts, not customersDetection works (question 638)
Review actions completedThe process learns from failures (question 455)

Signs of a process that only looks reliable:

  • Green runs, but people rerun failed jobs until they pass.
  • Successful deployments, followed by quiet manual fixes in production.
  • Nobody has rolled back in a year, so nobody knows whether it works.
  • A few experts deploy the important services by hand "to be safe".
  • Long gaps between releases, which hide problems instead of removing them.

The strongest single sign: releases are boring. Teams deploy during working hours, on any day, without a meeting, and when something goes wrong, it is caught early and undone in minutes.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet