Skip to content

Interview PrepKubernetes

Rollouts and the Pod Lifecycle ​

348. What is the difference between kubectl apply and kubectl create? ​

The short answer

kubectl create is imperative: it makes a new object and fails if one with that name already exists. kubectl apply is declarative: it creates the object if it is missing, and if it exists, it updates it to match your file, changing only the fields you manage and leaving fields set by others alone. apply can be run again and again with the same result, which is why pipelines and GitOps tools use it.

bash
kubectl create -f deploy.yaml     # first time: created
kubectl create -f deploy.yaml     # second time: Error from server (AlreadyExists)
kubectl apply  -f deploy.yaml     # created, or configured, or unchanged

How apply knows what to change. It compares three things: what you are applying now, what you applied last time, and what is live in the cluster. That lets it tell apart:

  • A field you removed from your file, which it should remove from the cluster.
  • A field someone else set, such as an autoscaler changing replicas, which it should leave alone if your file does not mention it.

Two ways it keeps that memory:

  1. Client-side apply, the traditional way, stores your last applied version in an annotation on the object, kubectl.kubernetes.io/last-applied-configuration.
  2. Server-side apply, kubectl apply --server-side, lets the API server track which field manager, such as you, a GitOps tool or an autoscaler, owns each field. If two managers try to set the same field differently, it reports a conflict instead of silently overwriting. Most GitOps tools use it.

Other commands in the family:

CommandBehaviour
createCreate only; fail if it exists
applyCreate or update to match the file
replaceOverwrite the whole object with the file
patchChange specific fields only
editOpen the live object in an editor, then save

A common trap: an object made with create, then later managed with apply, has no record of what was applied before, so the first apply cannot know which fields to remove. Pick apply from the start for anything that lasts.

349. How do you compare the manifest you intend to apply with the changes Kubernetes would make? ​

The short answer

Run kubectl diff -f on your files. It asks the API server to work out what would change, including defaults and changes made by admission webhooks, without saving anything, and prints the differences like a code review. For Helm charts, use the helm diff plugin. GitOps tools such as Argo CD show the same comparison in their interface before syncing. In CI, kubectl diff exits with code 1 when there are differences, so a pipeline can show or gate on them.

bash
kubectl diff -f deploy/prod/
diff
diff -u -N /tmp/LIVE-1532/apps.v1.Deployment.shop-prod.checkout-api /tmp/MERGED-1532/apps.v1.Deployment.shop-prod.checkout-api
--- /tmp/LIVE-1532/apps.v1.Deployment.shop-prod.checkout-api
+++ /tmp/MERGED-1532/apps.v1.Deployment.shop-prod.checkout-api
@@ -6,7 +6,7 @@
-  generation: 7
+  generation: 8
@@ -42,7 +42,7 @@
-        image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
+        image: myacr.azurecr.io/shop/checkout-api@sha256:9e07aa12...

What the exit code means: 0 means no differences, 1 means there are differences, and anything above 1 means an error.

Other ways to preview:

bash
kubectl apply -f deploy/prod/ --dry-run=server -o yaml     # (1)
helm diff upgrade checkout-api ./chart -f values-prod.yaml # (2)
  1. Shows the full objects exactly as the server would store them, with defaults filled in, without saving.
  2. The helm diff plugin shows what a Helm upgrade would change (question 505).

Why "server-side" matters. A client-only check, --dry-run=client, only knows your file. The server-side version runs the real admission chain, so you also see fields a webhook would inject, or a policy rejection, before the real change.

350. How do rolling updates work for a Deployment? ​

The short answer

When you change a Deployment's pod template, it creates a new ReplicaSet for the new version. Then it moves step by step. It starts some new pods and waits for them to become ready. Then it stops some old pods. It repeats this until every pod runs the new version. maxSurge and maxUnavailable decide how many pods change at each step. Old pods stop only as new ones become ready, so the service keeps serving throughout, as long as readiness probes are honest.

① start4 old, all serving② each stepstart 1 new (surge); when it is ready,stop 1 old; repeat③ done4 new, old ReplicaSet at 0maxSurge 1: at most 4 + 1 = 5 pods at any momentmaxUnavailable 0: never fewer than 4 ready pods
A rollout of four replicas. Blue: old version. Green: new version. With these settings, capacity never drops, and the rollout moves one pod at a time.

Step by step:

  1. You change the template, for example the image.
  2. The Deployment creates a new ReplicaSet with zero replicas.
  3. It scales the new ReplicaSet up, as far as maxSurge allows.
  4. Each new pod must pass its readiness probe. If minReadySeconds is set, it must also stay ready for that long before it counts as available.
  5. As new pods become available, the Deployment scales the old ReplicaSet down, as far as maxUnavailable allows.
  6. Steps 3 to 5 repeat until the new ReplicaSet has all the replicas and the old one has none.

What can stop it:

  • New pods never become ready. The rollout waits, old pods keep serving, and after progressDeadlineSeconds, 10 minutes by default, the Deployment reports that it is not progressing. It does not roll back by itself (question 428).
  • Not enough room in the cluster for the extra surge pods. They wait as Pending (question 408).

The hidden requirement: during the rollout, old and new versions serve users at the same time. They must be compatible with each other and with the database (question 286). If they cannot be, use a different strategy, such as blue-green (question 281).

351. What do maxSurge and maxUnavailable control during a rolling update? ​

The short answer

maxSurge is how many pods above the desired count may exist during a rollout, so it controls how fast new pods are added and how much extra capacity you need. maxUnavailable is how many pods below the desired count may be unavailable, so it controls how much capacity you may lose. Both default to 25%. Setting maxUnavailable to 0 guarantees full capacity throughout, and a higher maxSurge makes the rollout faster.

SettingMeaningDefaultRounding
maxSurgeExtra pods allowed above replicas25%Percentages round up
maxUnavailablePods allowed to be unavailable below replicas25%Percentages round down

They can be absolute numbers or percentages, and they cannot both be zero, since then nothing could ever change.

Worked examples with replicas: 4:

maxSurgemaxUnavailableMost pods at onceFewest ready podsBehaviour
25% = 125% = 153The default: quick, small dip
1054No capacity loss, one at a time
0143No extra capacity needed, a dip
100% = 4084Start all new at once, like a fast blue-green

With replicas: 10: 25% surge rounds up to 3 extra pods, and 25% unavailable rounds down to 2.

Choosing settings:

  • Busy production services: maxUnavailable: 0, so users never lose capacity, and a maxSurge that the cluster can afford, such as 25% or more.
  • Tight clusters with no spare room: maxSurge: 0 and maxUnavailable: 1, accepting a small dip.
  • Large deployments where speed matters: a higher maxSurge. Question 303 shows a 35-minute rollout shrinking to five minutes this way.

352. How do you check rollout status, pause or resume a rollout, and roll back a Deployment? ​

The short answer

kubectl rollout status waits for a rollout to finish and fails if it gets stuck. kubectl rollout pause and resume stop and restart a rollout, so you can batch several changes into one. kubectl rollout history lists the revisions, and kubectl rollout undo goes back to the previous revision, or --to-revision to a specific one. In a GitOps setup, roll back in Git instead, or the GitOps tool will put the newer version back.

bash
kubectl rollout status deploy/checkout-api -n shop-prod --timeout=5m   # (1)
kubectl rollout history deploy/checkout-api -n shop-prod               # (2)
kubectl rollout history deploy/checkout-api -n shop-prod --revision=7  # (3)
kubectl rollout pause deploy/checkout-api -n shop-prod                 # (4)
kubectl set image deploy/checkout-api api=myacr.azurecr.io/shop/checkout-api@sha256:9e07... -n shop-prod
kubectl set env deploy/checkout-api LOG_LEVEL=info -n shop-prod
kubectl rollout resume deploy/checkout-api -n shop-prod                # (5)
kubectl rollout undo deploy/checkout-api -n shop-prod                  # (6)
kubectl rollout undo deploy/checkout-api -n shop-prod --to-revision=5  # (7)
  1. Waits until the rollout completes, and exits with an error if it fails or exceeds five minutes. Pipelines use this (question 230).
  2. Lists revisions. The CHANGE-CAUSE column shows the kubernetes.io/change-cause annotation of each, if you set one.
  3. Shows the pod template of one revision.
  4. Pauses: template changes are recorded but no rollout starts.
  5. Resumes, and both changes roll out together as one revision.
  6. Goes back to the previous revision. Kubernetes scales up the old ReplicaSet and scales down the current one, as a normal rolling update.
  7. Goes back to a specific revision.
REVISION  CHANGE-CAUSE
5         deploy 1.4.0 from pipeline 9101
6         deploy 1.4.1 from pipeline 9120
7         deploy 1.4.2 from pipeline 9137

Two caveats:

  1. Rollback covers the pod template only. It does not undo changes to ConfigMaps, Secrets, Services or the database (question 448).
  2. GitOps fights manual rollbacks. If Argo CD or Flux manages the Deployment, rollout undo makes the cluster differ from Git, and the tool may restore the newer version. Revert the change in Git instead (question 512).

353. What does imagePullPolicy do, and how can mutable image tags create unexpected rollout behavior? ​

The short answer

imagePullPolicy tells the kubelet when to fetch an image. Always checks the registry every time a container starts, IfNotPresent uses the node's copy if it has one, and Never only uses the local copy. If you reuse a tag for different images, nodes that already have the old image keep running it with IfNotPresent, while Always can bring in a new version on any restart without a deployment. And pushing a new image under the same tag starts no rollout at all, because the Deployment did not change.

PolicyBefore starting a containerDefault when
AlwaysAsks the registry for the tag's current digest; downloads only if the node lacks itThe tag is latest, or there is no tag
IfNotPresentUses the local image if present; pulls only if missingAny other tag, or a digest
NeverUses the local image; fails if missingNever the default

Always is cheaper than it sounds. It does not download the whole image each time. It asks the registry which digest the tag points at, and only downloads layers the node does not already have. But it does make every container start depend on the registry being reachable.

What goes wrong with a reused tag, say checkout-api:prod:

  1. Nothing rolls out. You push a new image as :prod. The Deployment still says :prod, so its template is unchanged, and Kubernetes has no reason to replace any pod.
  2. Nodes disagree. With IfNotPresent, a pod restarting on a node that cached the old :prod keeps the old code, while a pod on a fresh node gets the new code. Same tag, two versions.
  3. Surprise releases. With Always, any pod that restarts for any reason, such as an OOM kill, quietly picks up whatever :prod means today, with no deployment, no approval and no record.

The fix: unique tags, or better, digests. image: ...@sha256:41c9... means exactly one image. Changing it is a real template change, so it starts a proper rollout, and every node runs the same bytes (question 87).

354. How do image pull credentials work when a Pod uses a private registry? ​

The short answer

The kubelet pulls images, so the credentials must reach the kubelet, not your app. The portable way is an imagePullSecrets entry on the pod, or on its service account, pointing at a Secret of type kubernetes.io/dockerconfigjson in the same namespace. On managed clouds, the better way is to give the nodes' own cloud identity permission to read the registry, for example by attaching an Azure Container Registry to an AKS cluster. Then there is no secret to store or rotate at all.

Option 1: a pull secret.

bash
kubectl create secret docker-registry acr-pull \
  --docker-server=myacr.azurecr.io \
  --docker-username=<token-name> \
  --docker-password=<token-password> \
  -n shop-prod
yaml
spec:
  imagePullSecrets:
    - name: acr-pull
  containers:
    - name: api
      image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...

Or attach it to the namespace's service account once, so every pod using that account gets it automatically:

bash
kubectl patch serviceaccount default -n shop-prod -p '{"imagePullSecrets":[{"name":"acr-pull"}]}'

Downsides: the secret must be copied into every namespace, and it is a long-lived credential that must be rotated.

Option 2: the node's cloud identity, best on managed Kubernetes:

CloudHow
Azure AKSaz aks update --attach-acr myacr gives the kubelet's managed identity the AcrPull role
AWS EKSThe node role gets permission to read from ECR
Google GKEThe nodes' service account gets read access to Artifact Registry

No secret exists anywhere. The kubelet gets short-lived tokens from the cloud by itself, through a component called a kubelet credential provider.

Keep pull and app credentials apart. Pull credentials let the node fetch images. They should be read-only and never reach the app's containers. The app's own credentials, such as its database password, are delivered separately (question 132).

355. What are init containers, sidecar containers, and ephemeral containers used for? ​

The short answer

Init containers run one after another to completion before the app containers start, to prepare things: wait for a dependency, fetch configuration, fix file permissions. Sidecar containers run alongside the app for its whole life, adding something such as log shipping or a proxy. Native sidecars, declared as init containers with restartPolicy: Always, start before the app and stop after it. Ephemeral containers are added to an already running pod, temporarily, for debugging.

Init containerSidecar containerEphemeral container
RunsBefore the app, to completion, in orderAlongside the appAdded to a running pod
Restarted if it failsYes, until it succeedsYesNo
PurposePreparationA helper for the app's whole lifeDebugging
Declared ininitContainersinitContainers with restartPolicy: Always, or containersAdded with kubectl debug
Can be removedFinishes by itselfWith the podNo, stays until the pod ends
yaml
spec:
  initContainers:
    - name: wait-for-db                                   # (1)
      image: busybox:1.36
      command: ["sh", "-c", "until nc -z orders-db 5432; do sleep 2; done"]
    - name: log-shipper                                   # (2)
      image: fluent/fluent-bit:3.1
      restartPolicy: Always
      volumeMounts: [{ name: logs, mountPath: /logs }]
  containers:
    - name: api                                           # (3)
      image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
      volumeMounts: [{ name: logs, mountPath: /logs }]
  volumes:
    - name: logs
      emptyDir: {}
  1. A classic init container: blocks until the database port answers, then exits. Only then does the next one start.
  2. A native sidecar: an init container with restartPolicy: Always. It starts before the app, keeps running alongside it, and is stopped only after the app has stopped, so no log lines are lost at shutdown. This became stable in Kubernetes 1.33.
  3. The app, which starts after both init containers have done their part.

Ephemeral containers for debugging:

bash
kubectl debug -it checkout-api-7d9f-abcde --image=busybox:1.36 --target=api

This adds a temporary busybox container to the running pod. --target puts it in the app container's process namespace, so you can see the app's processes. Question 414 covers this in depth.

356. How do termination grace periods and lifecycle hooks affect application shutdown? ​

The short answer

When a pod is deleted, Kubernetes first runs each container's preStop hook, if there is one, then sends SIGTERM, and finally SIGKILL if the container has not exited when the grace period ends. The grace period, terminationGracePeriodSeconds, defaults to 30 seconds and covers all of that, including the hook. A short preStop pause gives load balancers time to stop sending traffic. A postStart hook runs right after a container starts, but with no guarantee it runs before the app's own start.

yaml
spec:
  terminationGracePeriodSeconds: 45            # (1)
  containers:
    - name: api
      image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
      lifecycle:
        preStop:
          sleep: { seconds: 10 }               # (2)
  1. 45 seconds in total, from the delete request to SIGKILL.
  2. Pause for 10 seconds before SIGTERM. Newer Kubernetes versions support this built-in sleep action. On older ones, exec: { command: ["sleep", "10"] } does the same, but needs a sleep program in the image.

The order of events:

  1. The pod is marked for deletion. At the same moment, it is removed from its Services' endpoints (question 488).
  2. preStop runs: here, 10 seconds of waiting.
  3. SIGTERM is sent to the container's process 1.
  4. The app shuts down gracefully: finishes in-flight requests and closes connections (question 105).
  5. If it is still running when the 45 seconds are up, SIGKILL.

Why the preStop pause helps: removing the pod from endpoints takes a few seconds to reach every node and every load balancer. If the app stopped accepting connections at once, some requests would still arrive, and fail. The pause keeps the app serving normally until the rest of the system has stopped sending it traffic.

Getting the sum right. The grace period must cover the preStop time plus the app's own drain time. With a 10-second pause and an app that needs up to 20 seconds to drain, 30 seconds is too tight. 45 leaves a margin.

postStart runs as soon as the container is created, at the same time as its main process starts. Use it rarely, for small tasks such as registering somewhere. If it fails, the container is killed and restarted.

357. How should a one-off database migration be run in Kubernetes without tying it to every application replica startup? ​

The short answer

Run it once, as its own Kubernetes Job, before the new version of the application rolls out, either as a pipeline step, a Helm pre-upgrade hook, or an Argo CD pre-sync hook. Give the Job a retry limit and a deadline, use the same image as the application, and make the migrations safe to run again. Avoid running migrations in every replica's init container or start-up code, where many copies race each other and slow migrations delay every pod becoming ready.

Why not in every replica?

  1. Races. Ten pods starting at once means ten copies of the migration starting at once. Many migration tools take a lock, so nine wait, but some do not.
  2. Slow starts. A migration that takes three minutes makes every pod take three minutes to become ready, and during autoscaling or node failures that delay hurts.
  3. Wrong failure point. A failed migration shows up as crash-looping app pods, instead of one clearly failed migration step.

A migration Job:

yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: checkout-migrate-3f9c2ab
  namespace: shop-prod
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 900
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: checkout-migrator          # (1)
      containers:
        - name: migrate
          image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...   # (2)
          command: ["node", "dist/migrate.js"]
          envFrom: [{ secretRef: { name: checkout-migrator-db } }]       # (3)
  1. Its own identity, separate from the app's.
  2. The same image as the release, so the migrations match the code exactly.
  3. Database credentials with permission to change the schema, which the running app itself should not have.

Where to start it from:

OptionHow
Pipeline stepApply the Job, wait for completion, then roll out the app
Helm hookAnnotate the Job with helm.sh/hook: pre-upgrade
Argo CD hookAnnotate it with argocd.argoproj.io/hook: PreSync

And always: keep migrations backward compatible, so the old version keeps working while the new one rolls out (question 287).

488. What happens, step by step, when a Pod is deleted, and why can requests fail during a rollout? ​

The short answer

When a pod is deleted, two things start at the same moment, independently: the pod is removed from its Service's endpoints, and the kubelet begins shutting the pod down. The endpoint change takes a few seconds to reach every node's traffic rules, every ingress controller and every load balancer. If the app stops accepting connections immediately on SIGTERM, requests routed during those seconds fail. The fix is a short preStop pause, an app that keeps serving while it drains, and a grace period long enough for both.

deleteroutingendpoint removednodes and gateways catch upno more new trafficthe podpreStop: sleep 10s, still servingSIGTERMdrain, then exit 0the pod, without a pauseSIGTERMapp stopped, traffic still arrivingfailed requests, 502sTime flows to the right. Both timelines start at the same instant.
The shutdown race. Routing changes spread across the cluster in seconds, not instantly. The pause keeps the app answering until every router has stopped sending it new requests.

What happens, in order:

  1. The delete request arrives, for example from a rolling update or a scale-down. The API server sets the pod's deletionTimestamp. The pod shows as Terminating.
  2. Path A, routing: the EndpointSlice controller sees the pod is terminating and marks it as not ready in its Services' endpoints. Then each node's kube-proxy, or network plugin, rewrites its rules. Ingress controllers, gateways and cloud load balancers watching the endpoints update their own lists. Each of these takes its own time, from milliseconds to several seconds.
  3. Path B, the pod, at the same moment: the kubelet runs the preStop hook, then sends SIGTERM, then waits for the process to exit, up to the grace period, and finally sends SIGKILL if it must (question 356).

The race. Path A and path B are not coordinated. Without a pause, the app may receive SIGTERM and close its listening socket while some routers still believe it is a valid target. Requests sent there fail with "connection refused", or a 502 from the gateway. This is the most common cause of errors during otherwise healthy rollouts, and it is the story told in question 9.

The fix, in three parts:

  1. A preStop pause of a few seconds, often 5 to 15, so routing catches up before shutdown begins.
  2. A graceful app: on SIGTERM, stop accepting new connections, finish the ones in flight, then exit (question 105).
  3. A grace period that covers both, with a margin.

Also worth checking: that long-lived connections, such as keep-alive HTTP connections between the gateway and the pod, are closed by the app during drain, so the gateway opens fresh ones to other pods.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet