Appearance
Rollouts and the Pod Lifecycle
348. What is the difference between kubectl apply and kubectl create?
The short answer
kubectl create is imperative: it makes a new object and fails if one with that name already exists. kubectl apply is declarative: it creates the object if it is missing, and if it exists, it updates it to match your file, changing only the fields you manage and leaving fields set by others alone. apply can be run again and again with the same result, which is why pipelines and GitOps tools use it.
bash
kubectl create -f deploy.yaml # first time: created
kubectl create -f deploy.yaml # second time: Error from server (AlreadyExists)
kubectl apply -f deploy.yaml # created, or configured, or unchangedHow apply knows what to change. It compares three things: what you are applying now, what you applied last time, and what is live in the cluster. That lets it tell apart:
- A field you removed from your file, which it should remove from the cluster.
- A field someone else set, such as an autoscaler changing
replicas, which it should leave alone if your file does not mention it.
Two ways it keeps that memory:
- Client-side apply, the traditional way, stores your last applied version in an annotation on the object,
kubectl.kubernetes.io/last-applied-configuration. - Server-side apply,
kubectl apply --server-side, lets the API server track which field manager, such as you, a GitOps tool or an autoscaler, owns each field. If two managers try to set the same field differently, it reports a conflict instead of silently overwriting. Most GitOps tools use it.
Other commands in the family:
| Command | Behaviour |
|---|---|
create | Create only; fail if it exists |
apply | Create or update to match the file |
replace | Overwrite the whole object with the file |
patch | Change specific fields only |
edit | Open the live object in an editor, then save |
A common trap: an object made with create, then later managed with apply, has no record of what was applied before, so the first apply cannot know which fields to remove. Pick apply from the start for anything that lasts.
349. How do you compare the manifest you intend to apply with the changes Kubernetes would make?
The short answer
Run kubectl diff -f on your files. It asks the API server to work out what would change, including defaults and changes made by admission webhooks, without saving anything, and prints the differences like a code review. For Helm charts, use the helm diff plugin. GitOps tools such as Argo CD show the same comparison in their interface before syncing. In CI, kubectl diff exits with code 1 when there are differences, so a pipeline can show or gate on them.
bash
kubectl diff -f deploy/prod/diff
diff -u -N /tmp/LIVE-1532/apps.v1.Deployment.shop-prod.checkout-api /tmp/MERGED-1532/apps.v1.Deployment.shop-prod.checkout-api
--- /tmp/LIVE-1532/apps.v1.Deployment.shop-prod.checkout-api
+++ /tmp/MERGED-1532/apps.v1.Deployment.shop-prod.checkout-api
@@ -6,7 +6,7 @@
- generation: 7
+ generation: 8
@@ -42,7 +42,7 @@
- image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
+ image: myacr.azurecr.io/shop/checkout-api@sha256:9e07aa12...What the exit code means: 0 means no differences, 1 means there are differences, and anything above 1 means an error.
Other ways to preview:
bash
kubectl apply -f deploy/prod/ --dry-run=server -o yaml # (1)
helm diff upgrade checkout-api ./chart -f values-prod.yaml # (2)- Shows the full objects exactly as the server would store them, with defaults filled in, without saving.
- The
helm diffplugin shows what a Helm upgrade would change (question 505).
Why "server-side" matters. A client-only check, --dry-run=client, only knows your file. The server-side version runs the real admission chain, so you also see fields a webhook would inject, or a policy rejection, before the real change.
350. How do rolling updates work for a Deployment?
The short answer
When you change a Deployment's pod template, it creates a new ReplicaSet for the new version. Then it moves step by step. It starts some new pods and waits for them to become ready. Then it stops some old pods. It repeats this until every pod runs the new version. maxSurge and maxUnavailable decide how many pods change at each step. Old pods stop only as new ones become ready, so the service keeps serving throughout, as long as readiness probes are honest.
Step by step:
- You change the template, for example the image.
- The Deployment creates a new ReplicaSet with zero replicas.
- It scales the new ReplicaSet up, as far as
maxSurgeallows. - Each new pod must pass its readiness probe. If
minReadySecondsis set, it must also stay ready for that long before it counts as available. - As new pods become available, the Deployment scales the old ReplicaSet down, as far as
maxUnavailableallows. - Steps 3 to 5 repeat until the new ReplicaSet has all the replicas and the old one has none.
What can stop it:
- New pods never become ready. The rollout waits, old pods keep serving, and after
progressDeadlineSeconds, 10 minutes by default, the Deployment reports that it is not progressing. It does not roll back by itself (question 428). - Not enough room in the cluster for the extra surge pods. They wait as
Pending(question 408).
The hidden requirement: during the rollout, old and new versions serve users at the same time. They must be compatible with each other and with the database (question 286). If they cannot be, use a different strategy, such as blue-green (question 281).
351. What do maxSurge and maxUnavailable control during a rolling update?
The short answer
maxSurge is how many pods above the desired count may exist during a rollout, so it controls how fast new pods are added and how much extra capacity you need. maxUnavailable is how many pods below the desired count may be unavailable, so it controls how much capacity you may lose. Both default to 25%. Setting maxUnavailable to 0 guarantees full capacity throughout, and a higher maxSurge makes the rollout faster.
| Setting | Meaning | Default | Rounding |
|---|---|---|---|
maxSurge | Extra pods allowed above replicas | 25% | Percentages round up |
maxUnavailable | Pods allowed to be unavailable below replicas | 25% | Percentages round down |
They can be absolute numbers or percentages, and they cannot both be zero, since then nothing could ever change.
Worked examples with replicas: 4:
| maxSurge | maxUnavailable | Most pods at once | Fewest ready pods | Behaviour |
|---|---|---|---|---|
| 25% = 1 | 25% = 1 | 5 | 3 | The default: quick, small dip |
| 1 | 0 | 5 | 4 | No capacity loss, one at a time |
| 0 | 1 | 4 | 3 | No extra capacity needed, a dip |
| 100% = 4 | 0 | 8 | 4 | Start all new at once, like a fast blue-green |
With replicas: 10: 25% surge rounds up to 3 extra pods, and 25% unavailable rounds down to 2.
Choosing settings:
- Busy production services:
maxUnavailable: 0, so users never lose capacity, and amaxSurgethat the cluster can afford, such as 25% or more. - Tight clusters with no spare room:
maxSurge: 0andmaxUnavailable: 1, accepting a small dip. - Large deployments where speed matters: a higher
maxSurge. Question 303 shows a 35-minute rollout shrinking to five minutes this way.
352. How do you check rollout status, pause or resume a rollout, and roll back a Deployment?
The short answer
kubectl rollout status waits for a rollout to finish and fails if it gets stuck. kubectl rollout pause and resume stop and restart a rollout, so you can batch several changes into one. kubectl rollout history lists the revisions, and kubectl rollout undo goes back to the previous revision, or --to-revision to a specific one. In a GitOps setup, roll back in Git instead, or the GitOps tool will put the newer version back.
bash
kubectl rollout status deploy/checkout-api -n shop-prod --timeout=5m # (1)
kubectl rollout history deploy/checkout-api -n shop-prod # (2)
kubectl rollout history deploy/checkout-api -n shop-prod --revision=7 # (3)
kubectl rollout pause deploy/checkout-api -n shop-prod # (4)
kubectl set image deploy/checkout-api api=myacr.azurecr.io/shop/checkout-api@sha256:9e07... -n shop-prod
kubectl set env deploy/checkout-api LOG_LEVEL=info -n shop-prod
kubectl rollout resume deploy/checkout-api -n shop-prod # (5)
kubectl rollout undo deploy/checkout-api -n shop-prod # (6)
kubectl rollout undo deploy/checkout-api -n shop-prod --to-revision=5 # (7)- Waits until the rollout completes, and exits with an error if it fails or exceeds five minutes. Pipelines use this (question 230).
- Lists revisions. The
CHANGE-CAUSEcolumn shows thekubernetes.io/change-causeannotation of each, if you set one. - Shows the pod template of one revision.
- Pauses: template changes are recorded but no rollout starts.
- Resumes, and both changes roll out together as one revision.
- Goes back to the previous revision. Kubernetes scales up the old ReplicaSet and scales down the current one, as a normal rolling update.
- Goes back to a specific revision.
REVISION CHANGE-CAUSE
5 deploy 1.4.0 from pipeline 9101
6 deploy 1.4.1 from pipeline 9120
7 deploy 1.4.2 from pipeline 9137Two caveats:
- Rollback covers the pod template only. It does not undo changes to ConfigMaps, Secrets, Services or the database (question 448).
- GitOps fights manual rollbacks. If Argo CD or Flux manages the Deployment,
rollout undomakes the cluster differ from Git, and the tool may restore the newer version. Revert the change in Git instead (question 512).
353. What does imagePullPolicy do, and how can mutable image tags create unexpected rollout behavior?
The short answer
imagePullPolicy tells the kubelet when to fetch an image. Always checks the registry every time a container starts, IfNotPresent uses the node's copy if it has one, and Never only uses the local copy. If you reuse a tag for different images, nodes that already have the old image keep running it with IfNotPresent, while Always can bring in a new version on any restart without a deployment. And pushing a new image under the same tag starts no rollout at all, because the Deployment did not change.
| Policy | Before starting a container | Default when |
|---|---|---|
Always | Asks the registry for the tag's current digest; downloads only if the node lacks it | The tag is latest, or there is no tag |
IfNotPresent | Uses the local image if present; pulls only if missing | Any other tag, or a digest |
Never | Uses the local image; fails if missing | Never the default |
Always is cheaper than it sounds. It does not download the whole image each time. It asks the registry which digest the tag points at, and only downloads layers the node does not already have. But it does make every container start depend on the registry being reachable.
What goes wrong with a reused tag, say checkout-api:prod:
- Nothing rolls out. You push a new image as
:prod. The Deployment still says:prod, so its template is unchanged, and Kubernetes has no reason to replace any pod. - Nodes disagree. With
IfNotPresent, a pod restarting on a node that cached the old:prodkeeps the old code, while a pod on a fresh node gets the new code. Same tag, two versions. - Surprise releases. With
Always, any pod that restarts for any reason, such as an OOM kill, quietly picks up whatever:prodmeans today, with no deployment, no approval and no record.
The fix: unique tags, or better, digests. image: ...@sha256:41c9... means exactly one image. Changing it is a real template change, so it starts a proper rollout, and every node runs the same bytes (question 87).
354. How do image pull credentials work when a Pod uses a private registry?
The short answer
The kubelet pulls images, so the credentials must reach the kubelet, not your app. The portable way is an imagePullSecrets entry on the pod, or on its service account, pointing at a Secret of type kubernetes.io/dockerconfigjson in the same namespace. On managed clouds, the better way is to give the nodes' own cloud identity permission to read the registry, for example by attaching an Azure Container Registry to an AKS cluster. Then there is no secret to store or rotate at all.
Option 1: a pull secret.
bash
kubectl create secret docker-registry acr-pull \
--docker-server=myacr.azurecr.io \
--docker-username=<token-name> \
--docker-password=<token-password> \
-n shop-prodyaml
spec:
imagePullSecrets:
- name: acr-pull
containers:
- name: api
image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...Or attach it to the namespace's service account once, so every pod using that account gets it automatically:
bash
kubectl patch serviceaccount default -n shop-prod -p '{"imagePullSecrets":[{"name":"acr-pull"}]}'Downsides: the secret must be copied into every namespace, and it is a long-lived credential that must be rotated.
Option 2: the node's cloud identity, best on managed Kubernetes:
| Cloud | How |
|---|---|
| Azure AKS | az aks update --attach-acr myacr gives the kubelet's managed identity the AcrPull role |
| AWS EKS | The node role gets permission to read from ECR |
| Google GKE | The nodes' service account gets read access to Artifact Registry |
No secret exists anywhere. The kubelet gets short-lived tokens from the cloud by itself, through a component called a kubelet credential provider.
Keep pull and app credentials apart. Pull credentials let the node fetch images. They should be read-only and never reach the app's containers. The app's own credentials, such as its database password, are delivered separately (question 132).
355. What are init containers, sidecar containers, and ephemeral containers used for?
The short answer
Init containers run one after another to completion before the app containers start, to prepare things: wait for a dependency, fetch configuration, fix file permissions. Sidecar containers run alongside the app for its whole life, adding something such as log shipping or a proxy. Native sidecars, declared as init containers with restartPolicy: Always, start before the app and stop after it. Ephemeral containers are added to an already running pod, temporarily, for debugging.
| Init container | Sidecar container | Ephemeral container | |
|---|---|---|---|
| Runs | Before the app, to completion, in order | Alongside the app | Added to a running pod |
| Restarted if it fails | Yes, until it succeeds | Yes | No |
| Purpose | Preparation | A helper for the app's whole life | Debugging |
| Declared in | initContainers | initContainers with restartPolicy: Always, or containers | Added with kubectl debug |
| Can be removed | Finishes by itself | With the pod | No, stays until the pod ends |
yaml
spec:
initContainers:
- name: wait-for-db # (1)
image: busybox:1.36
command: ["sh", "-c", "until nc -z orders-db 5432; do sleep 2; done"]
- name: log-shipper # (2)
image: fluent/fluent-bit:3.1
restartPolicy: Always
volumeMounts: [{ name: logs, mountPath: /logs }]
containers:
- name: api # (3)
image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
volumeMounts: [{ name: logs, mountPath: /logs }]
volumes:
- name: logs
emptyDir: {}- A classic init container: blocks until the database port answers, then exits. Only then does the next one start.
- A native sidecar: an init container with
restartPolicy: Always. It starts before the app, keeps running alongside it, and is stopped only after the app has stopped, so no log lines are lost at shutdown. This became stable in Kubernetes 1.33. - The app, which starts after both init containers have done their part.
Ephemeral containers for debugging:
bash
kubectl debug -it checkout-api-7d9f-abcde --image=busybox:1.36 --target=apiThis adds a temporary busybox container to the running pod. --target puts it in the app container's process namespace, so you can see the app's processes. Question 414 covers this in depth.
356. How do termination grace periods and lifecycle hooks affect application shutdown?
The short answer
When a pod is deleted, Kubernetes first runs each container's preStop hook, if there is one, then sends SIGTERM, and finally SIGKILL if the container has not exited when the grace period ends. The grace period, terminationGracePeriodSeconds, defaults to 30 seconds and covers all of that, including the hook. A short preStop pause gives load balancers time to stop sending traffic. A postStart hook runs right after a container starts, but with no guarantee it runs before the app's own start.
yaml
spec:
terminationGracePeriodSeconds: 45 # (1)
containers:
- name: api
image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
lifecycle:
preStop:
sleep: { seconds: 10 } # (2)- 45 seconds in total, from the delete request to
SIGKILL. - Pause for 10 seconds before
SIGTERM. Newer Kubernetes versions support this built-insleepaction. On older ones,exec: { command: ["sleep", "10"] }does the same, but needs asleepprogram in the image.
The order of events:
- The pod is marked for deletion. At the same moment, it is removed from its Services' endpoints (question 488).
preStopruns: here, 10 seconds of waiting.SIGTERMis sent to the container's process 1.- The app shuts down gracefully: finishes in-flight requests and closes connections (question 105).
- If it is still running when the 45 seconds are up,
SIGKILL.
Why the preStop pause helps: removing the pod from endpoints takes a few seconds to reach every node and every load balancer. If the app stopped accepting connections at once, some requests would still arrive, and fail. The pause keeps the app serving normally until the rest of the system has stopped sending it traffic.
Getting the sum right. The grace period must cover the preStop time plus the app's own drain time. With a 10-second pause and an app that needs up to 20 seconds to drain, 30 seconds is too tight. 45 leaves a margin.
postStart runs as soon as the container is created, at the same time as its main process starts. Use it rarely, for small tasks such as registering somewhere. If it fails, the container is killed and restarted.
357. How should a one-off database migration be run in Kubernetes without tying it to every application replica startup?
The short answer
Run it once, as its own Kubernetes Job, before the new version of the application rolls out, either as a pipeline step, a Helm pre-upgrade hook, or an Argo CD pre-sync hook. Give the Job a retry limit and a deadline, use the same image as the application, and make the migrations safe to run again. Avoid running migrations in every replica's init container or start-up code, where many copies race each other and slow migrations delay every pod becoming ready.
Why not in every replica?
- Races. Ten pods starting at once means ten copies of the migration starting at once. Many migration tools take a lock, so nine wait, but some do not.
- Slow starts. A migration that takes three minutes makes every pod take three minutes to become ready, and during autoscaling or node failures that delay hurts.
- Wrong failure point. A failed migration shows up as crash-looping app pods, instead of one clearly failed migration step.
A migration Job:
yaml
apiVersion: batch/v1
kind: Job
metadata:
name: checkout-migrate-3f9c2ab
namespace: shop-prod
spec:
backoffLimit: 2
activeDeadlineSeconds: 900
ttlSecondsAfterFinished: 86400
template:
spec:
restartPolicy: Never
serviceAccountName: checkout-migrator # (1)
containers:
- name: migrate
image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8... # (2)
command: ["node", "dist/migrate.js"]
envFrom: [{ secretRef: { name: checkout-migrator-db } }] # (3)- Its own identity, separate from the app's.
- The same image as the release, so the migrations match the code exactly.
- Database credentials with permission to change the schema, which the running app itself should not have.
Where to start it from:
| Option | How |
|---|---|
| Pipeline step | Apply the Job, wait for completion, then roll out the app |
| Helm hook | Annotate the Job with helm.sh/hook: pre-upgrade |
| Argo CD hook | Annotate it with argocd.argoproj.io/hook: PreSync |
And always: keep migrations backward compatible, so the old version keeps working while the new one rolls out (question 287).
488. What happens, step by step, when a Pod is deleted, and why can requests fail during a rollout?
The short answer
When a pod is deleted, two things start at the same moment, independently: the pod is removed from its Service's endpoints, and the kubelet begins shutting the pod down. The endpoint change takes a few seconds to reach every node's traffic rules, every ingress controller and every load balancer. If the app stops accepting connections immediately on SIGTERM, requests routed during those seconds fail. The fix is a short preStop pause, an app that keeps serving while it drains, and a grace period long enough for both.
What happens, in order:
- The delete request arrives, for example from a rolling update or a scale-down. The API server sets the pod's
deletionTimestamp. The pod shows asTerminating. - Path A, routing: the EndpointSlice controller sees the pod is terminating and marks it as not ready in its Services' endpoints. Then each node's kube-proxy, or network plugin, rewrites its rules. Ingress controllers, gateways and cloud load balancers watching the endpoints update their own lists. Each of these takes its own time, from milliseconds to several seconds.
- Path B, the pod, at the same moment: the kubelet runs the
preStophook, then sendsSIGTERM, then waits for the process to exit, up to the grace period, and finally sendsSIGKILLif it must (question 356).
The race. Path A and path B are not coordinated. Without a pause, the app may receive SIGTERM and close its listening socket while some routers still believe it is a valid target. Requests sent there fail with "connection refused", or a 502 from the gateway. This is the most common cause of errors during otherwise healthy rollouts, and it is the story told in question 9.
The fix, in three parts:
- A
preStoppause of a few seconds, often 5 to 15, so routing catches up before shutdown begins. - A graceful app: on
SIGTERM, stop accepting new connections, finish the ones in flight, then exit (question 105). - A grace period that covers both, with a margin.
Also worth checking: that long-lived connections, such as keep-alive HTTP connections between the gateway and the pod, are closed by the app during drain, so the gateway opens fresh ones to other pods.