Skip to content

Interview PrepKubernetes

Probes and Troubleshooting ​

401. What is the difference between startup, readiness, and liveness probes? ​

The short answer

They are three health checks the kubelet runs against a container, each answering a different question. The startup probe asks "has it finished starting yet?", and holds the other two back until it passes. The readiness probe asks "can it serve traffic right now?", and a failure removes the pod from its Services without restarting it. The liveness probe asks "is it still alive, or stuck?", and a failure makes the kubelet restart the container.

ProbeQuestionOn failureRuns
StartupFinished starting?Keep waiting; restart after failureThresholdOnly until it first succeeds
ReadinessReady for traffic now?Removed from Service endpointsAll the time
LivenessAlive, not stuck?Container restartedAll the time, after startup passes
yaml
startupProbe:
  httpGet: { path: /healthz, port: http }
  periodSeconds: 2
  failureThreshold: 60          # (1)
readinessProbe:
  httpGet: { path: /ready, port: http }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2           # (2)
livenessProbe:
  httpGet: { path: /healthz, port: http }
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3           # (3)
  1. Allows up to 2 × 60 = 120 seconds to start. Liveness and readiness do not run until this passes, so a slow start is never mistaken for a hang.
  2. Two failed checks in a row, about 10 seconds, take the pod out of traffic.
  3. Three failed checks in a row, about 30 seconds, restart the container.

The four ways to probe:

TypePasses when
httpGetThe response status is between 200 and 399
tcpSocketA TCP connection to the port opens
grpcThe standard gRPC health check answers "serving"
execA command inside the container exits with 0

The defaults, when you leave fields out: checked every 10 seconds, a 1-second timeout, 3 failures to count as failed, and 1 success to count as passing again. The 1-second timeout is often too short for a busy app (question 404).

402. What happens when a readiness probe fails, and what happens when a liveness probe fails? ​

The short answer

When readiness fails, the pod stays running, but it is marked not ready, removed from its Services' endpoints and from load balancers, so it receives no new traffic. When the probe passes again, it is added back. When liveness fails, the kubelet kills the container and starts it again, which counts as a restart. Repeated liveness failures lead to growing delays between restarts, shown as CrashLoopBackOff.

readiness failspod keeps running, marked not readyremoved from Service endpoints: no trafficprobe passes again: traffic returnsliveness failskubelet kills the containercontainer restarts; restart count + 1keeps failing: CrashLoopBackOff
Two different reactions. A readiness failure is gentle and reversible: the pod rests out of traffic. A liveness failure is drastic: the container is thrown away and started again.

After a readiness failure:

  1. The kubelet sets the pod's Ready condition to false.
  2. The EndpointSlice controller marks the pod as not ready, and every node's Service rules stop sending it new traffic.
  3. Connections already open may continue, depending on the client and proxies.
  4. The probe keeps running. When it passes successThreshold times, the pod is ready again and traffic returns.

After a liveness failure:

  1. The kubelet stops the container: it runs any preStop hook, sends SIGTERM, waits for the grace period, then sends SIGKILL (question 356).
  2. It starts the container again, according to the pod's restart policy. Anything held only in the container's memory is lost.
  3. The event "Liveness probe failed" and the rising restart count show in kubectl describe pod.
  4. If it fails over and over, restarts are spaced out more and more, which shows as CrashLoopBackOff (question 407).

The rule of thumb: readiness is for "busy, starting up or temporarily unable", which resolves by itself. Liveness is only for "stuck forever", which only a restart can fix.

403. How can a badly configured liveness probe make an overloaded service less reliable? ​

The short answer

If the liveness probe fails whenever the service is merely slow, overload turns into restarts. Under heavy load, health checks time out, the kubelet restarts the busiest pods, and the remaining pods get even more traffic, so they slow down and get restarted too. If the probe also checks a shared dependency, such as the database, one slow database restarts every pod at once. A liveness probe should only detect a truly stuck process, with generous timeouts, and never check dependencies.

How the spiral works:

  1. Traffic spikes. Pods are busy, and responses, including health check responses, slow down.
  2. The liveness probe has a 1-second timeout. Health checks start timing out.
  3. After three failures, the kubelet restarts the container. Its in-flight requests fail, its caches are lost, and while it restarts it serves nothing.
  4. Its share of traffic moves to the other pods, which are now even busier, so their probes time out too.
  5. More restarts, less capacity, more load per pod. The service collapses, even though no pod was actually broken.

The dependency version is worse. If /healthz checks the database, and the database becomes slow for 30 seconds, every pod fails liveness at the same moment. Kubernetes restarts the entire service at once, and thousands of requests fail, because of a database blip that would otherwise have passed.

How to write liveness probes that help instead of hurt:

  1. Check only the process itself. Is the event loop responding? Is the main thread alive? No database, no outside calls.
  2. Generous timeouts and thresholds, such as a 3 to 5 second timeout and 3 to 6 failures, so slowness is not mistaken for death.
  3. Use readiness for load and dependencies. If the app cannot serve because it is overloaded or a dependency is down, failing readiness takes it out of traffic without destroying it.
  4. Consider no liveness probe at all. If your app crashes cleanly when it breaks, the kubelet restarts it anyway. A liveness probe only adds value for apps that can hang without crashing.

404. How would you choose probe paths, ports, delays, timeouts, and failure thresholds for a web service? ​

The short answer

Use separate, purpose-built endpoints: a light one for liveness that only proves the process works, and one for readiness that says whether the app can serve right now. Use a startup probe to cover slow starts instead of a long initial delay. Set timeouts comfortably above how long the endpoint takes under load, often 2 to 5 seconds. Choose periods and thresholds by deciding how fast you want to react: detection time is roughly the period multiplied by the failure threshold.

Endpoints:

EndpointUsed byChecks
/healthzLiveness, and startupThe process responds; no outside calls
/readyReadinessStarted and warmed up, not shutting down, critical dependencies reachable

A reasonable set of values, with the reasoning:

yaml
startupProbe:
  httpGet: { path: /healthz, port: http }
  periodSeconds: 3
  failureThreshold: 40      # up to 120 s to start: covers the slowest cold start seen, plus margin
readinessProbe:
  httpGet: { path: /ready, port: http }
  periodSeconds: 5
  timeoutSeconds: 2         # well above /ready's slowest normal response
  failureThreshold: 2       # out of traffic within about 10 s
  successThreshold: 1
livenessProbe:
  httpGet: { path: /healthz, port: http }
  periodSeconds: 10
  timeoutSeconds: 5         # tolerant of garbage-collection pauses under load
  failureThreshold: 6       # restart only after about 60 s of real unresponsiveness

Decisions explained:

  1. Startup probe instead of initialDelaySeconds. A fixed delay must cover the slowest start, which slows every start. A startup probe passes as soon as the app is up.
  2. Detection time = period × failure threshold. Readiness should react within seconds. Liveness can wait a minute, because a wrong restart is more harmful than a slow one.
  3. Timeouts above the slow end of the endpoint's real response times under load, not its average.
  4. The port. Probing the app's main port proves the main server answers. A separate admin port is fine, as long as it is served by the same process and would also stop answering if the app hung.
  5. Readiness during shutdown. When the app receives SIGTERM, it can start failing /ready on purpose, which helps drain traffic (question 488).

Check them for real. Watch probe failures during a load test. Probes that fail under normal peak traffic will cause the spiral in question 403.

405. What is the difference between Pending, Running, Succeeded, Failed, Unknown, and CrashLoopBackOff Pod states or conditions? ​

The short answer

Pending, Running, Succeeded, Failed and Unknown are a pod's phases, its overall stage. Pending means accepted but not yet running all its containers, for example still being scheduled or pulling images. Running means it is on a node and at least one container is running. Succeeded and Failed mean all containers have ended, all successfully or not. Unknown means the node stopped reporting. CrashLoopBackOff is not a phase: it is a container's waiting reason, meaning it keeps crashing and the kubelet is waiting longer before each restart.

Pendingscheduling, pullingRunningCrashLoopBackOffa container's reasonSucceeded: all exited 0Failed: one exited badlyUnknown: node silent
Phases and a reason. Phases describe the whole pod. CrashLoopBackOff lives inside a Running pod, describing one container that keeps failing. Jobs end in Succeeded or Failed; long-running services stay in Running.
Shown asWhat it isMeaning
PendingPhaseNot scheduled yet, or containers not all started: pulling images, creating volumes
RunningPhaseOn a node; at least one container running, starting or restarting
SucceededPhaseAll containers ended with exit code 0, and will not restart
FailedPhaseAll containers ended, at least one with an error, or the pod was evicted
UnknownPhaseThe pod's node is not reporting
CrashLoopBackOffContainer waiting reasonThe container keeps crashing; restarts are delayed
ImagePullBackOffContainer waiting reasonThe image cannot be pulled; retries are delayed
ContainerCreatingContainer waiting reasonSetting up: pulling, mounting, networking

The STATUS column of kubectl get pods shows the most useful of these, which is why phases and reasons appear mixed together there.

The back-off. After each crash, the kubelet waits longer before the next restart: 10 seconds, then 20, 40 and so on, up to five minutes. If a container then runs for 10 minutes without crashing, the delay resets. This protects the node from a container restarting in a tight loop.

Pod conditions give finer detail, each true or false: PodScheduled, Initialized when init containers are done, ContainersReady, and Ready, which is what Services use. kubectl describe pod lists them.

406. What does ImagePullBackOff usually mean, and how do you troubleshoot it? ​

The short answer

It means the kubelet tried to pull the container's image, failed, and is now waiting longer before each retry. The usual causes are a wrong image name or tag, missing or expired registry credentials, a registry the node cannot reach, rate limits on a public registry, or no image built for the node's processor type. The exact cause is in the pod's events: kubectl describe pod shows the registry's error message.

bash
kubectl describe pod checkout-api-5c8b-x2k9 -n shop-prod | sed -n '/Events/,$p'
Warning  Failed     12s  kubelet  Failed to pull image "myacr.azurecr.io/shop/checkout-api:3f9c2ab": rpc error: code = NotFound desc = failed to pull and unpack image "myacr.azurecr.io/shop/checkout-api:3f9c2ab": not found
Warning  Failed     12s  kubelet  Error: ErrImagePull
Normal   BackOff    1s   kubelet  Back-off pulling image "myacr.azurecr.io/shop/checkout-api:3f9c2ab"

Read the message, then match it:

Message containsLikely causeCheck
not found, manifest unknownWrong name or tag, or never pushedThe exact tag in the registry; did the push finish?
401, unauthorized, deniedMissing or expired credentialsPull secret or node identity (question 354)
no such host, i/o timeoutRegistry unreachableNode DNS, firewall, private endpoint
toomanyrequests, 429Rate limitDocker Hub limits (question 477)
no match for platform in manifestNo image for this CPU typeBuild for arm64 or amd64 (question 82)
context deadline exceededVery slow pullImage size, network (question 160)

Remember the trap from question 160: many registries answer "unauthorized" for an image that does not exist, so check the name before assuming a permission problem.

Testing from the node's point of view. The pull happens on the node, with the node's identity and network. A pull from your laptop working proves little. On managed clusters, a debug pod on the same node, or the cloud's own diagnostics, such as az aks check-acr, test from the right place.

Once fixed, nothing else is needed: the kubelet's next retry pulls the image and the container starts. Deleting the pod makes it retry immediately.

407. What does CrashLoopBackOff indicate, and which commands would you run first? ​

The short answer

It means the container starts, then exits or is killed, again and again, and the kubelet is waiting longer and longer between restarts. The app itself is failing, so look at why it exits. First kubectl describe pod for the last exit code, the reason and the events, then kubectl logs --previous for what the crashed instance printed. The exit code and the last log lines explain most cases: missing configuration, a failing dependency, too little memory, a wrong command, or a liveness probe killing it.

bash
kubectl get pod checkout-api-5c8b-x2k9 -n shop-prod                       # (1)
kubectl describe pod checkout-api-5c8b-x2k9 -n shop-prod                  # (2)
kubectl logs checkout-api-5c8b-x2k9 -n shop-prod --previous               # (3)
kubectl logs checkout-api-5c8b-x2k9 -n shop-prod -c api --previous        # (4)
  1. The restart count and how long ago it last restarted.
  2. Look at Last State: the reason and exit code of the previous run, and the Events, such as "Liveness probe failed" or "Back-off restarting failed container".
  3. The log of the previous run, the one that crashed. The current run's log is often empty, because it has only just started.
  4. The same for one container, in a pod with several.

Match the evidence:

EvidenceLikely cause
Exit code 1, an error in the logsApp error: missing setting, bad configuration, failed connection
Reason OOMKilled, exit code 137Over its memory limit (question 411)
Exit code 127, "not found"Wrong command or entrypoint
Exit code 126, "permission denied"Script not executable, or a non-root user cannot read a file
Events show "Liveness probe failed"The probe kills it (question 403)
Exit code 0, then restartsThe program finishes, but the pod's restart policy is Always
"read-only file system" in logsreadOnlyRootFilesystem with no writable emptyDir where the app writes

When the logs show nothing useful, start a copy of the pod that does not run the real command, so you can look around:

bash
kubectl debug checkout-api-5c8b-x2k9 -n shop-prod -it --copy-to=checkout-debug --container=api -- sh

This creates a new pod, checkout-debug, copied from the original, with the api container's command replaced by a shell. You can then inspect its files and environment and run the real command by hand.

408. How do you investigate a Pod stuck in Pending? ​

The short answer

First see whether it has been scheduled to a node. If not, the scheduler is telling you why, in the pod's events. The message looks like "0/5 nodes are available", followed by the reasons. There may not be enough CPU or memory for its requests. No node may match its selectors or affinity. There may be taints it does not tolerate, volume problems, or spreading rules that cannot be met. If it has a node but is still Pending, it is stuck before its containers start, usually pulling an image or mounting a volume.

Step 1: scheduled or not?

bash
kubectl get pod checkout-api-5c8b-x2k9 -n shop-prod -o wide

An empty NODE column means not scheduled yet. A node name means it is stuck after scheduling.

Step 2a: not scheduled. The events say why:

bash
kubectl describe pod checkout-api-5c8b-x2k9 -n shop-prod | grep -A 3 FailedScheduling
Warning  FailedScheduling  30s  default-scheduler  0/5 nodes are available: 2 Insufficient memory, 1 node(s) had untolerated taint {dedicated: gpu}, 2 node(s) didn't match pod topology spread constraints.
Reason in the messageFix
Insufficient cpu or Insufficient memoryLower requests, add nodes, or let the autoscaler add them
didn't match Pod's node affinity/selectorFix the labels or the selector
had untolerated taintAdd a toleration, or use other nodes
didn't match pod anti-affinity rulesMore nodes, or soften the rule (question 395)
didn't match pod topology spread constraintsMore domains, or ScheduleAnyway (question 396)
pod has unbound immediate PersistentVolumeClaimsFix the storage claim (question 384)
volume node affinity conflictThe disk is in another zone than the available nodes
Too many podsThe node reached its maximum pod count

If a node autoscaler is installed, also check why it did not add a node: it may have reached its maximum, hit a cloud quota, or found that no node type could fit the pod. Its events or logs say so.

Step 2b: scheduled, but still Pending. Look at the container status. ContainerCreating shows with events about pulling images (question 406), mounting volumes, or setting up networking. A networking message such as "failed to assign an IP address" can mean the node's subnet has run out of addresses (question 614).

409. How do you investigate a Pod that is Running but not Ready? ​

The short answer

The containers are running, but at least one is failing its readiness probe, so the pod gets no traffic. Find which container and why: the events show "Readiness probe failed" with the status code or error. Then call the readiness endpoint yourself from inside the pod. The app may still be warming up. A dependency the readiness check needs may be unreachable. The probe path or port may be wrong. The app may listen only on 127.0.0.1. The timeout may be too short. Or a sidecar container may not be ready.

bash
kubectl get pod web-6f5c-abcde -n shop-prod                               # (1)
kubectl describe pod web-6f5c-abcde -n shop-prod | grep -E "Ready|Readiness"   # (2)
kubectl exec -n shop-prod web-6f5c-abcde -c web -- wget -qO- -T 3 http://localhost:8080/ready   # (3)
kubectl logs -n shop-prod web-6f5c-abcde -c web --tail=50                 # (4)
  1. READY 1/2 shows how many containers are ready.
  2. Which containers are not ready, and the probe failure messages, such as Readiness probe failed: HTTP probe failed with statuscode: 503, or context deadline exceeded, which means a timeout.
  3. Calls the endpoint from inside the pod, as the kubelet does. If the image has no tools, use a debug container (question 414).
  4. The app's own explanation, often a log line saying what the readiness check is waiting for.

Common causes:

ClueCause
Returns 503 with "warming up"Still starting; add or lengthen a startup probe
Returns 503, a dependency named in the logsThe readiness check requires a dependency that is down
Connection refusedWrong port, or the app listens on 127.0.0.1 only
404Wrong path in the probe
TimeoutsThe endpoint is slow; raise timeoutSeconds or make the check lighter
App container ready, pod still notAnother container, such as a sidecar, is not ready
All containers ready, pod still notA readiness gate is unmet, such as a cloud load balancer reporting the target unhealthy

Is "not ready" actually correct? Sometimes the probe is right: the app really cannot serve, because its database is down. Then the fix is the dependency, and readiness is doing exactly its job by keeping traffic away (question 451).

410. How do you use kubectl describe, events, current logs, and previous-container logs to troubleshoot a failure? ​

The short answer

Use them in a fixed order, from broad to detailed. kubectl get shows the status and the restart count. kubectl describe shows the container states, including how the last run ended, the pod's conditions, and recent events, which explain most scheduling, image, probe and kill problems. kubectl get events shows everything happening in the namespace in time order. kubectl logs shows what the app is saying now, and kubectl logs --previous shows what it said just before it last crashed.

The order, and what each step answers:

StepCommandAnswers
1kubectl get pods -o wideWhich pods, which state, how many restarts, which node
2kubectl describe pod NAMEWhy it is in that state: last exit, conditions, events
3kubectl get events --sort-by=.lastTimestampWhat else happened around it
4kubectl logs NAME --previousWhat the crashed run printed
5kubectl logs NAME -fWhat the current run prints

Reading describe well. Three parts matter most:

text
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
    Restart Count:  6
Conditions:
  Ready             False
Events:
  Warning  BackOff   2m (x25 over 8m)  kubelet  Back-off restarting failed container api
  • State is what the container is doing now.
  • Last State is how the previous run ended, which is usually the real clue.
  • Events are the timeline: scheduling, pulls, starts, probe failures, kills. (x25 over 8m) means it happened 25 times in 8 minutes.

Two cautions about events:

  1. They are kept only for a short time, one hour by default, so look early, or have a tool that copies them into your log store.
  2. They belong to objects. A pod that was never created, because of a quota or a policy, has no events of its own: look at its ReplicaSet's events instead (question 428).

Useful log options: -c NAME for one container, --all-containers, --since=10m, --timestamps, and -l app=checkout-api --max-log-requests=10 to read several pods at once.

411. How would you diagnose OOMKilled, and what would you change only after identifying the cause? ​

The short answer

Confirm it first: the container's last state says OOMKilled with exit code 137. Then find out why its memory reached the limit. It may be a real leak that grows over time. It may be a spike caused by particular work, such as a big request or batch. The language runtime may size its memory wrongly for the container. Memory outside the main heap and memory-backed volumes count too. Or the limit may simply be too low for normal use. Only once you know which, change the right thing: the code, the runtime settings, the work size, or the limit and request together.

Step 1: confirm. kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137 (question 391).

Step 2: look at memory over time for that container, from your monitoring, up to each kill:

  • Steady climb, a sawtooth over hours: a leak (question 156).
  • Sudden jump: a spike caused by specific work.
  • A flat line just under the limit: normal use is simply above what the limit allows.

Step 3: check the usual causes:

CauseHow to recognise it
A leak in the codeGrowth that never levels off; heap snapshots show one kind of object growing
Spikes from big workKills follow large uploads, reports or batch sizes
Runtime sized for the wrong limitThe heap setting is close to, or above, the container limit
Memory outside the heapThread stacks, native libraries, buffers: the process is much bigger than the heap
Memory-backed volumesAn emptyDir with medium: Memory filling up
SidecarsAnother container in the pod also uses memory, but each has its own limit
The limit is simply too lowStable use, killed at start-up or at peak

About runtimes. A Node.js process sizes its main heap automatically, and modern versions take the container's limit into account. But the heap is only part of the process's memory. So a setting such as --max-old-space-size should leave room below the limit. The Java virtual machine reads the container limit and, by default, uses a quarter of it for the heap. -XX:MaxRAMPercentage=75 gives it more, while still leaving room for everything else (question 68).

Step 4: change one thing, for the right reason:

  • A leak: fix the code. Raising the limit only stretches the time between crashes.
  • Spikes: limit the work, such as smaller batches, streaming instead of loading whole files, or fewer requests handled at once.
  • Runtime settings: align the heap with the limit.
  • A limit that is too low: raise the limit and the request, so the scheduler reserves what the pod really needs.

412. How would you investigate a Pod that keeps restarting only under production traffic? ​

The short answer

Find out how it dies each time: killed for memory, killed by a failing liveness probe, or exiting with an error. Then line those deaths up with traffic and with resource metrics. Under load, there are a few usual causes. Liveness probes time out because the app is busy or throttled. Memory grows with the number of requests in progress. Connection pools or thread pools run dry, so the health endpoint hangs. Dependencies time out or fail, and the code does not handle it. Or the app hits operating system limits, such as open files.

Step 1: how does it die? kubectl describe pod and its events, for each restart:

EvidencePoints to
OOMKilledMemory grows with load
"Liveness probe failed", then a killThe probe fails under load
Exit code 1 with a stack traceAn unhandled error under load
Exit code 137 without OOMKilledKilled after a failed liveness probe, or by something else (question 151)

Step 2: line it up with load. Plot restarts against requests per second, CPU, memory, throttling and garbage collection time.

Step 3: the usual load-only causes:

  1. The liveness probe times out. At peak, the app is busy, or CPU-throttled (question 392), or paused for garbage collection, and the health endpoint takes longer than its timeout. The kubelet restarts a perfectly working pod (question 403).
  2. Memory per request. Each request in progress holds memory. Double the concurrent requests, double that memory, and the limit is reached only at peak.
  3. Exhausted pools. All database connections are in use. Requests, including health checks if they touch the pool, queue up and time out.
  4. Unhandled dependency failures. A downstream API starts failing under load, and an unhandled error in the code crashes the process.
  5. Operating system limits. Too many open connections hit the open-file limit, or ephemeral ports run out.

Step 4: reproduce it outside production. A load test in staging with production-like traffic usually reproduces it within minutes, where you can watch, profile and change things freely.

Step 5: fix the cause, and also make the pod more forgiving: realistic probe timeouts, bounded concurrency, timeouts and retries on dependencies, and enough replicas, or autoscaling, so no single pod carries a peak alone (question 452).

413. How would you debug a container when it has no shell or debugging tools installed? ​

The short answer

Bring the tools from outside instead of putting them in the image. Attach an ephemeral debug container with a tools image to the running pod, sharing its process namespace, so you can see and inspect the app's processes and files. Or make a copy of the pod with a different image or command. For node-level problems, start a debug pod on the node itself. Logs, port-forwarding and the cluster's own metrics also work with no tools in the image at all.

Option 1: an ephemeral debug container, the main tool (question 414):

bash
kubectl debug -it -n shop-prod checkout-api-5c8b-x2k9 --image=nicolaka/netshoot --target=api

You get a shell in a new container, inside the same pod, with network tools such as curl, dig and tcpdump, sharing the pod's network and, because of --target, the api container's processes. The app's files are visible at /proc/1/root/, where 1 is the app's process number as seen from the shared namespace.

Option 2: a modified copy of the pod:

bash
kubectl debug -n shop-prod checkout-api-5c8b-x2k9 --copy-to=checkout-debug \
  --set-image=api=myacr.azurecr.io/shop/checkout-api-debug@sha256:5b1e... -it -- sh

The copy uses a debug variant of the image, with a shell. Useful when the original pod crashes too quickly to attach anything to it. Many distroless images publish :debug variants for exactly this.

Option 3: the node:

bash
kubectl debug node/aks-general-12345678-vmss000003 -it --image=ubuntu

This runs a pod on that node with access to the node's namespaces and its filesystem mounted at /host. From there, chroot /host lets you use the node's own tools, such as crictl to inspect containers and journalctl for the kubelet's logs.

Without any shell at all:

  • kubectl logs, kubectl describe, events and metrics.
  • kubectl port-forward to call the app's endpoints from your laptop (question 490).

Not usable: kubectl cp relies on tar inside the container, so it fails on images that do not have it.

414. How do ephemeral debug containers help investigate a running Pod? ​

The short answer

An ephemeral container is a temporary container added to a pod that is already running, without restarting it or changing its spec. It can use any image, so you bring your own tools. It shares the pod's network, and can share a target container's process namespace, so you can see the app's processes, open connections and files exactly as they are, at the moment of the problem. It cannot be removed and is never restarted, and it disappears when the pod does.

How it works:

  1. kubectl debug asks the API server to add a container to the pod's ephemeralContainers list, through a special subresource.
  2. The kubelet starts it inside the existing pod sandbox, so it joins the pod's network namespace at once.
  3. With --target=api, it also joins the api container's process namespace.
bash
kubectl debug -it -n shop-prod checkout-api-5c8b-x2k9 --image=busybox:1.36 --target=api
# inside the debug container:
ps                          # (1)
ls /proc/1/root/app         # (2)
netstat -tn                 # (3)
cat /proc/1/environ | tr '\0' '\n'   # (4)
  1. The app's processes are visible, because the process namespace is shared.
  2. The app container's filesystem, seen through its main process.
  3. The pod's open network connections.
  4. The environment the app was started with.

What makes them different from normal containers:

Normal containerEphemeral container
AddedWhen the pod is createdTo a running pod
Restarts the podChanging containers recreates the podNo
Ports, probes, resource settingsAllowedNot allowed
Restarted if it exitsAccording to policyNever
RemovedWith the podOnly with the pod

Security. Adding one needs permission on the pods/ephemeralcontainers subresource, which is powerful: it gives a shell inside any pod in the namespace, with its network and, often, its secrets. Grant it carefully in production. kubectl debug --profile chooses security settings for the debug container, such as restricted to satisfy strict pod security rules, or netadmin when you need packet capture.

498. How do you troubleshoot a node in NotReady state? ​

The short answer

NotReady means the control plane no longer believes the node is healthy: either its kubelet stopped sending heartbeats, or the kubelet itself reports a problem, such as the container runtime being down or the network plugin not ready. Check the node's conditions and events, then the kubelet and the container runtime on the machine itself, its disk, memory and process count, its network path to the API server, and its certificates. On managed clusters, the provider can often repair or replace the node automatically.

bash
kubectl get nodes
kubectl describe node aks-general-12345678-vmss000003
Conditions:
  Type             Status   Reason                       Message
  MemoryPressure   False    KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True     KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure      False    KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False    KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady

What NotReady means for pods. The node gets taints such as node.kubernetes.io/not-ready or node.kubernetes.io/unreachable. New pods are not scheduled there, and existing pods are evicted after their toleration time, five minutes by default, so their controllers recreate them elsewhere.

Then, on the node itself, through SSH, the cloud's console, or kubectl debug node/... if the node is still partly reachable:

bash
systemctl status kubelet
journalctl -u kubelet --since "30 min ago" | tail -50
systemctl status containerd
crictl ps
df -h; free -m

The usual causes:

CauseTypical sign
The kubelet stopped or keeps crashingsystemctl status kubelet shows failed; its log says why
The container runtime is down"container runtime is down" in the condition
The network plugin is not readyNetworkPluginNotReady; its DaemonSet pod is failing on this node
Full diskDiskPressure; images and logs filled it
Memory or process exhaustionMemoryPressure, PIDPressure, or the node too overloaded to respond
No network to the API serverHeartbeats stop; the node shows Unknown conditions
Expired kubelet certificatesErrors about certificates in the kubelet log
The virtual machine itself is gone or stuckThe cloud console shows it stopped or failing

On managed Kubernetes, node auto-repair features, in AKS, EKS and GKE, restart or reimage unhealthy nodes automatically. Often the fastest fix is to drain the node, if possible, and let the node pool replace it, then investigate the cause from logs afterwards (question 426).

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet