Appearance
Probes and Troubleshooting
401. What is the difference between startup, readiness, and liveness probes?
The short answer
They are three health checks the kubelet runs against a container, each answering a different question. The startup probe asks "has it finished starting yet?", and holds the other two back until it passes. The readiness probe asks "can it serve traffic right now?", and a failure removes the pod from its Services without restarting it. The liveness probe asks "is it still alive, or stuck?", and a failure makes the kubelet restart the container.
| Probe | Question | On failure | Runs |
|---|---|---|---|
| Startup | Finished starting? | Keep waiting; restart after failureThreshold | Only until it first succeeds |
| Readiness | Ready for traffic now? | Removed from Service endpoints | All the time |
| Liveness | Alive, not stuck? | Container restarted | All the time, after startup passes |
yaml
startupProbe:
httpGet: { path: /healthz, port: http }
periodSeconds: 2
failureThreshold: 60 # (1)
readinessProbe:
httpGet: { path: /ready, port: http }
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2 # (2)
livenessProbe:
httpGet: { path: /healthz, port: http }
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3 # (3)- Allows up to 2 × 60 = 120 seconds to start. Liveness and readiness do not run until this passes, so a slow start is never mistaken for a hang.
- Two failed checks in a row, about 10 seconds, take the pod out of traffic.
- Three failed checks in a row, about 30 seconds, restart the container.
The four ways to probe:
| Type | Passes when |
|---|---|
httpGet | The response status is between 200 and 399 |
tcpSocket | A TCP connection to the port opens |
grpc | The standard gRPC health check answers "serving" |
exec | A command inside the container exits with 0 |
The defaults, when you leave fields out: checked every 10 seconds, a 1-second timeout, 3 failures to count as failed, and 1 success to count as passing again. The 1-second timeout is often too short for a busy app (question 404).
402. What happens when a readiness probe fails, and what happens when a liveness probe fails?
The short answer
When readiness fails, the pod stays running, but it is marked not ready, removed from its Services' endpoints and from load balancers, so it receives no new traffic. When the probe passes again, it is added back. When liveness fails, the kubelet kills the container and starts it again, which counts as a restart. Repeated liveness failures lead to growing delays between restarts, shown as CrashLoopBackOff.
After a readiness failure:
- The kubelet sets the pod's
Readycondition to false. - The EndpointSlice controller marks the pod as not ready, and every node's Service rules stop sending it new traffic.
- Connections already open may continue, depending on the client and proxies.
- The probe keeps running. When it passes
successThresholdtimes, the pod is ready again and traffic returns.
After a liveness failure:
- The kubelet stops the container: it runs any
preStophook, sendsSIGTERM, waits for the grace period, then sendsSIGKILL(question 356). - It starts the container again, according to the pod's restart policy. Anything held only in the container's memory is lost.
- The event "Liveness probe failed" and the rising restart count show in
kubectl describe pod. - If it fails over and over, restarts are spaced out more and more, which shows as
CrashLoopBackOff(question 407).
The rule of thumb: readiness is for "busy, starting up or temporarily unable", which resolves by itself. Liveness is only for "stuck forever", which only a restart can fix.
403. How can a badly configured liveness probe make an overloaded service less reliable?
The short answer
If the liveness probe fails whenever the service is merely slow, overload turns into restarts. Under heavy load, health checks time out, the kubelet restarts the busiest pods, and the remaining pods get even more traffic, so they slow down and get restarted too. If the probe also checks a shared dependency, such as the database, one slow database restarts every pod at once. A liveness probe should only detect a truly stuck process, with generous timeouts, and never check dependencies.
How the spiral works:
- Traffic spikes. Pods are busy, and responses, including health check responses, slow down.
- The liveness probe has a 1-second timeout. Health checks start timing out.
- After three failures, the kubelet restarts the container. Its in-flight requests fail, its caches are lost, and while it restarts it serves nothing.
- Its share of traffic moves to the other pods, which are now even busier, so their probes time out too.
- More restarts, less capacity, more load per pod. The service collapses, even though no pod was actually broken.
The dependency version is worse. If /healthz checks the database, and the database becomes slow for 30 seconds, every pod fails liveness at the same moment. Kubernetes restarts the entire service at once, and thousands of requests fail, because of a database blip that would otherwise have passed.
How to write liveness probes that help instead of hurt:
- Check only the process itself. Is the event loop responding? Is the main thread alive? No database, no outside calls.
- Generous timeouts and thresholds, such as a 3 to 5 second timeout and 3 to 6 failures, so slowness is not mistaken for death.
- Use readiness for load and dependencies. If the app cannot serve because it is overloaded or a dependency is down, failing readiness takes it out of traffic without destroying it.
- Consider no liveness probe at all. If your app crashes cleanly when it breaks, the kubelet restarts it anyway. A liveness probe only adds value for apps that can hang without crashing.
404. How would you choose probe paths, ports, delays, timeouts, and failure thresholds for a web service?
The short answer
Use separate, purpose-built endpoints: a light one for liveness that only proves the process works, and one for readiness that says whether the app can serve right now. Use a startup probe to cover slow starts instead of a long initial delay. Set timeouts comfortably above how long the endpoint takes under load, often 2 to 5 seconds. Choose periods and thresholds by deciding how fast you want to react: detection time is roughly the period multiplied by the failure threshold.
Endpoints:
| Endpoint | Used by | Checks |
|---|---|---|
/healthz | Liveness, and startup | The process responds; no outside calls |
/ready | Readiness | Started and warmed up, not shutting down, critical dependencies reachable |
A reasonable set of values, with the reasoning:
yaml
startupProbe:
httpGet: { path: /healthz, port: http }
periodSeconds: 3
failureThreshold: 40 # up to 120 s to start: covers the slowest cold start seen, plus margin
readinessProbe:
httpGet: { path: /ready, port: http }
periodSeconds: 5
timeoutSeconds: 2 # well above /ready's slowest normal response
failureThreshold: 2 # out of traffic within about 10 s
successThreshold: 1
livenessProbe:
httpGet: { path: /healthz, port: http }
periodSeconds: 10
timeoutSeconds: 5 # tolerant of garbage-collection pauses under load
failureThreshold: 6 # restart only after about 60 s of real unresponsivenessDecisions explained:
- Startup probe instead of
initialDelaySeconds. A fixed delay must cover the slowest start, which slows every start. A startup probe passes as soon as the app is up. - Detection time = period × failure threshold. Readiness should react within seconds. Liveness can wait a minute, because a wrong restart is more harmful than a slow one.
- Timeouts above the slow end of the endpoint's real response times under load, not its average.
- The port. Probing the app's main port proves the main server answers. A separate admin port is fine, as long as it is served by the same process and would also stop answering if the app hung.
- Readiness during shutdown. When the app receives
SIGTERM, it can start failing/readyon purpose, which helps drain traffic (question 488).
Check them for real. Watch probe failures during a load test. Probes that fail under normal peak traffic will cause the spiral in question 403.
405. What is the difference between Pending, Running, Succeeded, Failed, Unknown, and CrashLoopBackOff Pod states or conditions?
The short answer
Pending, Running, Succeeded, Failed and Unknown are a pod's phases, its overall stage. Pending means accepted but not yet running all its containers, for example still being scheduled or pulling images. Running means it is on a node and at least one container is running. Succeeded and Failed mean all containers have ended, all successfully or not. Unknown means the node stopped reporting. CrashLoopBackOff is not a phase: it is a container's waiting reason, meaning it keeps crashing and the kubelet is waiting longer before each restart.
| Shown as | What it is | Meaning |
|---|---|---|
Pending | Phase | Not scheduled yet, or containers not all started: pulling images, creating volumes |
Running | Phase | On a node; at least one container running, starting or restarting |
Succeeded | Phase | All containers ended with exit code 0, and will not restart |
Failed | Phase | All containers ended, at least one with an error, or the pod was evicted |
Unknown | Phase | The pod's node is not reporting |
CrashLoopBackOff | Container waiting reason | The container keeps crashing; restarts are delayed |
ImagePullBackOff | Container waiting reason | The image cannot be pulled; retries are delayed |
ContainerCreating | Container waiting reason | Setting up: pulling, mounting, networking |
The STATUS column of kubectl get pods shows the most useful of these, which is why phases and reasons appear mixed together there.
The back-off. After each crash, the kubelet waits longer before the next restart: 10 seconds, then 20, 40 and so on, up to five minutes. If a container then runs for 10 minutes without crashing, the delay resets. This protects the node from a container restarting in a tight loop.
Pod conditions give finer detail, each true or false: PodScheduled, Initialized when init containers are done, ContainersReady, and Ready, which is what Services use. kubectl describe pod lists them.
406. What does ImagePullBackOff usually mean, and how do you troubleshoot it?
The short answer
It means the kubelet tried to pull the container's image, failed, and is now waiting longer before each retry. The usual causes are a wrong image name or tag, missing or expired registry credentials, a registry the node cannot reach, rate limits on a public registry, or no image built for the node's processor type. The exact cause is in the pod's events: kubectl describe pod shows the registry's error message.
bash
kubectl describe pod checkout-api-5c8b-x2k9 -n shop-prod | sed -n '/Events/,$p'Warning Failed 12s kubelet Failed to pull image "myacr.azurecr.io/shop/checkout-api:3f9c2ab": rpc error: code = NotFound desc = failed to pull and unpack image "myacr.azurecr.io/shop/checkout-api:3f9c2ab": not found
Warning Failed 12s kubelet Error: ErrImagePull
Normal BackOff 1s kubelet Back-off pulling image "myacr.azurecr.io/shop/checkout-api:3f9c2ab"Read the message, then match it:
| Message contains | Likely cause | Check |
|---|---|---|
not found, manifest unknown | Wrong name or tag, or never pushed | The exact tag in the registry; did the push finish? |
401, unauthorized, denied | Missing or expired credentials | Pull secret or node identity (question 354) |
no such host, i/o timeout | Registry unreachable | Node DNS, firewall, private endpoint |
toomanyrequests, 429 | Rate limit | Docker Hub limits (question 477) |
no match for platform in manifest | No image for this CPU type | Build for arm64 or amd64 (question 82) |
context deadline exceeded | Very slow pull | Image size, network (question 160) |
Remember the trap from question 160: many registries answer "unauthorized" for an image that does not exist, so check the name before assuming a permission problem.
Testing from the node's point of view. The pull happens on the node, with the node's identity and network. A pull from your laptop working proves little. On managed clusters, a debug pod on the same node, or the cloud's own diagnostics, such as az aks check-acr, test from the right place.
Once fixed, nothing else is needed: the kubelet's next retry pulls the image and the container starts. Deleting the pod makes it retry immediately.
407. What does CrashLoopBackOff indicate, and which commands would you run first?
The short answer
It means the container starts, then exits or is killed, again and again, and the kubelet is waiting longer and longer between restarts. The app itself is failing, so look at why it exits. First kubectl describe pod for the last exit code, the reason and the events, then kubectl logs --previous for what the crashed instance printed. The exit code and the last log lines explain most cases: missing configuration, a failing dependency, too little memory, a wrong command, or a liveness probe killing it.
bash
kubectl get pod checkout-api-5c8b-x2k9 -n shop-prod # (1)
kubectl describe pod checkout-api-5c8b-x2k9 -n shop-prod # (2)
kubectl logs checkout-api-5c8b-x2k9 -n shop-prod --previous # (3)
kubectl logs checkout-api-5c8b-x2k9 -n shop-prod -c api --previous # (4)- The restart count and how long ago it last restarted.
- Look at Last State: the reason and exit code of the previous run, and the Events, such as "Liveness probe failed" or "Back-off restarting failed container".
- The log of the previous run, the one that crashed. The current run's log is often empty, because it has only just started.
- The same for one container, in a pod with several.
Match the evidence:
| Evidence | Likely cause |
|---|---|
| Exit code 1, an error in the logs | App error: missing setting, bad configuration, failed connection |
Reason OOMKilled, exit code 137 | Over its memory limit (question 411) |
| Exit code 127, "not found" | Wrong command or entrypoint |
| Exit code 126, "permission denied" | Script not executable, or a non-root user cannot read a file |
| Events show "Liveness probe failed" | The probe kills it (question 403) |
| Exit code 0, then restarts | The program finishes, but the pod's restart policy is Always |
| "read-only file system" in logs | readOnlyRootFilesystem with no writable emptyDir where the app writes |
When the logs show nothing useful, start a copy of the pod that does not run the real command, so you can look around:
bash
kubectl debug checkout-api-5c8b-x2k9 -n shop-prod -it --copy-to=checkout-debug --container=api -- shThis creates a new pod, checkout-debug, copied from the original, with the api container's command replaced by a shell. You can then inspect its files and environment and run the real command by hand.
408. How do you investigate a Pod stuck in Pending?
The short answer
First see whether it has been scheduled to a node. If not, the scheduler is telling you why, in the pod's events. The message looks like "0/5 nodes are available", followed by the reasons. There may not be enough CPU or memory for its requests. No node may match its selectors or affinity. There may be taints it does not tolerate, volume problems, or spreading rules that cannot be met. If it has a node but is still Pending, it is stuck before its containers start, usually pulling an image or mounting a volume.
Step 1: scheduled or not?
bash
kubectl get pod checkout-api-5c8b-x2k9 -n shop-prod -o wideAn empty NODE column means not scheduled yet. A node name means it is stuck after scheduling.
Step 2a: not scheduled. The events say why:
bash
kubectl describe pod checkout-api-5c8b-x2k9 -n shop-prod | grep -A 3 FailedSchedulingWarning FailedScheduling 30s default-scheduler 0/5 nodes are available: 2 Insufficient memory, 1 node(s) had untolerated taint {dedicated: gpu}, 2 node(s) didn't match pod topology spread constraints.| Reason in the message | Fix |
|---|---|
Insufficient cpu or Insufficient memory | Lower requests, add nodes, or let the autoscaler add them |
didn't match Pod's node affinity/selector | Fix the labels or the selector |
had untolerated taint | Add a toleration, or use other nodes |
didn't match pod anti-affinity rules | More nodes, or soften the rule (question 395) |
didn't match pod topology spread constraints | More domains, or ScheduleAnyway (question 396) |
pod has unbound immediate PersistentVolumeClaims | Fix the storage claim (question 384) |
volume node affinity conflict | The disk is in another zone than the available nodes |
Too many pods | The node reached its maximum pod count |
If a node autoscaler is installed, also check why it did not add a node: it may have reached its maximum, hit a cloud quota, or found that no node type could fit the pod. Its events or logs say so.
Step 2b: scheduled, but still Pending. Look at the container status. ContainerCreating shows with events about pulling images (question 406), mounting volumes, or setting up networking. A networking message such as "failed to assign an IP address" can mean the node's subnet has run out of addresses (question 614).
409. How do you investigate a Pod that is Running but not Ready?
The short answer
The containers are running, but at least one is failing its readiness probe, so the pod gets no traffic. Find which container and why: the events show "Readiness probe failed" with the status code or error. Then call the readiness endpoint yourself from inside the pod. The app may still be warming up. A dependency the readiness check needs may be unreachable. The probe path or port may be wrong. The app may listen only on 127.0.0.1. The timeout may be too short. Or a sidecar container may not be ready.
bash
kubectl get pod web-6f5c-abcde -n shop-prod # (1)
kubectl describe pod web-6f5c-abcde -n shop-prod | grep -E "Ready|Readiness" # (2)
kubectl exec -n shop-prod web-6f5c-abcde -c web -- wget -qO- -T 3 http://localhost:8080/ready # (3)
kubectl logs -n shop-prod web-6f5c-abcde -c web --tail=50 # (4)READY 1/2shows how many containers are ready.- Which containers are not ready, and the probe failure messages, such as
Readiness probe failed: HTTP probe failed with statuscode: 503, orcontext deadline exceeded, which means a timeout. - Calls the endpoint from inside the pod, as the kubelet does. If the image has no tools, use a debug container (question 414).
- The app's own explanation, often a log line saying what the readiness check is waiting for.
Common causes:
| Clue | Cause |
|---|---|
| Returns 503 with "warming up" | Still starting; add or lengthen a startup probe |
| Returns 503, a dependency named in the logs | The readiness check requires a dependency that is down |
| Connection refused | Wrong port, or the app listens on 127.0.0.1 only |
| 404 | Wrong path in the probe |
| Timeouts | The endpoint is slow; raise timeoutSeconds or make the check lighter |
| App container ready, pod still not | Another container, such as a sidecar, is not ready |
| All containers ready, pod still not | A readiness gate is unmet, such as a cloud load balancer reporting the target unhealthy |
Is "not ready" actually correct? Sometimes the probe is right: the app really cannot serve, because its database is down. Then the fix is the dependency, and readiness is doing exactly its job by keeping traffic away (question 451).
410. How do you use kubectl describe, events, current logs, and previous-container logs to troubleshoot a failure?
The short answer
Use them in a fixed order, from broad to detailed. kubectl get shows the status and the restart count. kubectl describe shows the container states, including how the last run ended, the pod's conditions, and recent events, which explain most scheduling, image, probe and kill problems. kubectl get events shows everything happening in the namespace in time order. kubectl logs shows what the app is saying now, and kubectl logs --previous shows what it said just before it last crashed.
The order, and what each step answers:
| Step | Command | Answers |
|---|---|---|
| 1 | kubectl get pods -o wide | Which pods, which state, how many restarts, which node |
| 2 | kubectl describe pod NAME | Why it is in that state: last exit, conditions, events |
| 3 | kubectl get events --sort-by=.lastTimestamp | What else happened around it |
| 4 | kubectl logs NAME --previous | What the crashed run printed |
| 5 | kubectl logs NAME -f | What the current run prints |
Reading describe well. Three parts matter most:
text
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Restart Count: 6
Conditions:
Ready False
Events:
Warning BackOff 2m (x25 over 8m) kubelet Back-off restarting failed container api- State is what the container is doing now.
- Last State is how the previous run ended, which is usually the real clue.
- Events are the timeline: scheduling, pulls, starts, probe failures, kills.
(x25 over 8m)means it happened 25 times in 8 minutes.
Two cautions about events:
- They are kept only for a short time, one hour by default, so look early, or have a tool that copies them into your log store.
- They belong to objects. A pod that was never created, because of a quota or a policy, has no events of its own: look at its ReplicaSet's events instead (question 428).
Useful log options: -c NAME for one container, --all-containers, --since=10m, --timestamps, and -l app=checkout-api --max-log-requests=10 to read several pods at once.
411. How would you diagnose OOMKilled, and what would you change only after identifying the cause?
The short answer
Confirm it first: the container's last state says OOMKilled with exit code 137. Then find out why its memory reached the limit. It may be a real leak that grows over time. It may be a spike caused by particular work, such as a big request or batch. The language runtime may size its memory wrongly for the container. Memory outside the main heap and memory-backed volumes count too. Or the limit may simply be too low for normal use. Only once you know which, change the right thing: the code, the runtime settings, the work size, or the limit and request together.
Step 1: confirm. kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137 (question 391).
Step 2: look at memory over time for that container, from your monitoring, up to each kill:
- Steady climb, a sawtooth over hours: a leak (question 156).
- Sudden jump: a spike caused by specific work.
- A flat line just under the limit: normal use is simply above what the limit allows.
Step 3: check the usual causes:
| Cause | How to recognise it |
|---|---|
| A leak in the code | Growth that never levels off; heap snapshots show one kind of object growing |
| Spikes from big work | Kills follow large uploads, reports or batch sizes |
| Runtime sized for the wrong limit | The heap setting is close to, or above, the container limit |
| Memory outside the heap | Thread stacks, native libraries, buffers: the process is much bigger than the heap |
| Memory-backed volumes | An emptyDir with medium: Memory filling up |
| Sidecars | Another container in the pod also uses memory, but each has its own limit |
| The limit is simply too low | Stable use, killed at start-up or at peak |
About runtimes. A Node.js process sizes its main heap automatically, and modern versions take the container's limit into account. But the heap is only part of the process's memory. So a setting such as --max-old-space-size should leave room below the limit. The Java virtual machine reads the container limit and, by default, uses a quarter of it for the heap. -XX:MaxRAMPercentage=75 gives it more, while still leaving room for everything else (question 68).
Step 4: change one thing, for the right reason:
- A leak: fix the code. Raising the limit only stretches the time between crashes.
- Spikes: limit the work, such as smaller batches, streaming instead of loading whole files, or fewer requests handled at once.
- Runtime settings: align the heap with the limit.
- A limit that is too low: raise the limit and the request, so the scheduler reserves what the pod really needs.
412. How would you investigate a Pod that keeps restarting only under production traffic?
The short answer
Find out how it dies each time: killed for memory, killed by a failing liveness probe, or exiting with an error. Then line those deaths up with traffic and with resource metrics. Under load, there are a few usual causes. Liveness probes time out because the app is busy or throttled. Memory grows with the number of requests in progress. Connection pools or thread pools run dry, so the health endpoint hangs. Dependencies time out or fail, and the code does not handle it. Or the app hits operating system limits, such as open files.
Step 1: how does it die? kubectl describe pod and its events, for each restart:
| Evidence | Points to |
|---|---|
OOMKilled | Memory grows with load |
| "Liveness probe failed", then a kill | The probe fails under load |
| Exit code 1 with a stack trace | An unhandled error under load |
Exit code 137 without OOMKilled | Killed after a failed liveness probe, or by something else (question 151) |
Step 2: line it up with load. Plot restarts against requests per second, CPU, memory, throttling and garbage collection time.
Step 3: the usual load-only causes:
- The liveness probe times out. At peak, the app is busy, or CPU-throttled (question 392), or paused for garbage collection, and the health endpoint takes longer than its timeout. The kubelet restarts a perfectly working pod (question 403).
- Memory per request. Each request in progress holds memory. Double the concurrent requests, double that memory, and the limit is reached only at peak.
- Exhausted pools. All database connections are in use. Requests, including health checks if they touch the pool, queue up and time out.
- Unhandled dependency failures. A downstream API starts failing under load, and an unhandled error in the code crashes the process.
- Operating system limits. Too many open connections hit the open-file limit, or ephemeral ports run out.
Step 4: reproduce it outside production. A load test in staging with production-like traffic usually reproduces it within minutes, where you can watch, profile and change things freely.
Step 5: fix the cause, and also make the pod more forgiving: realistic probe timeouts, bounded concurrency, timeouts and retries on dependencies, and enough replicas, or autoscaling, so no single pod carries a peak alone (question 452).
413. How would you debug a container when it has no shell or debugging tools installed?
The short answer
Bring the tools from outside instead of putting them in the image. Attach an ephemeral debug container with a tools image to the running pod, sharing its process namespace, so you can see and inspect the app's processes and files. Or make a copy of the pod with a different image or command. For node-level problems, start a debug pod on the node itself. Logs, port-forwarding and the cluster's own metrics also work with no tools in the image at all.
Option 1: an ephemeral debug container, the main tool (question 414):
bash
kubectl debug -it -n shop-prod checkout-api-5c8b-x2k9 --image=nicolaka/netshoot --target=apiYou get a shell in a new container, inside the same pod, with network tools such as curl, dig and tcpdump, sharing the pod's network and, because of --target, the api container's processes. The app's files are visible at /proc/1/root/, where 1 is the app's process number as seen from the shared namespace.
Option 2: a modified copy of the pod:
bash
kubectl debug -n shop-prod checkout-api-5c8b-x2k9 --copy-to=checkout-debug \
--set-image=api=myacr.azurecr.io/shop/checkout-api-debug@sha256:5b1e... -it -- shThe copy uses a debug variant of the image, with a shell. Useful when the original pod crashes too quickly to attach anything to it. Many distroless images publish :debug variants for exactly this.
Option 3: the node:
bash
kubectl debug node/aks-general-12345678-vmss000003 -it --image=ubuntuThis runs a pod on that node with access to the node's namespaces and its filesystem mounted at /host. From there, chroot /host lets you use the node's own tools, such as crictl to inspect containers and journalctl for the kubelet's logs.
Without any shell at all:
kubectl logs,kubectl describe, events and metrics.kubectl port-forwardto call the app's endpoints from your laptop (question 490).
Not usable: kubectl cp relies on tar inside the container, so it fails on images that do not have it.
414. How do ephemeral debug containers help investigate a running Pod?
The short answer
An ephemeral container is a temporary container added to a pod that is already running, without restarting it or changing its spec. It can use any image, so you bring your own tools. It shares the pod's network, and can share a target container's process namespace, so you can see the app's processes, open connections and files exactly as they are, at the moment of the problem. It cannot be removed and is never restarted, and it disappears when the pod does.
How it works:
kubectl debugasks the API server to add a container to the pod'sephemeralContainerslist, through a special subresource.- The kubelet starts it inside the existing pod sandbox, so it joins the pod's network namespace at once.
- With
--target=api, it also joins theapicontainer's process namespace.
bash
kubectl debug -it -n shop-prod checkout-api-5c8b-x2k9 --image=busybox:1.36 --target=api
# inside the debug container:
ps # (1)
ls /proc/1/root/app # (2)
netstat -tn # (3)
cat /proc/1/environ | tr '\0' '\n' # (4)- The app's processes are visible, because the process namespace is shared.
- The app container's filesystem, seen through its main process.
- The pod's open network connections.
- The environment the app was started with.
What makes them different from normal containers:
| Normal container | Ephemeral container | |
|---|---|---|
| Added | When the pod is created | To a running pod |
| Restarts the pod | Changing containers recreates the pod | No |
| Ports, probes, resource settings | Allowed | Not allowed |
| Restarted if it exits | According to policy | Never |
| Removed | With the pod | Only with the pod |
Security. Adding one needs permission on the pods/ephemeralcontainers subresource, which is powerful: it gives a shell inside any pod in the namespace, with its network and, often, its secrets. Grant it carefully in production. kubectl debug --profile chooses security settings for the debug container, such as restricted to satisfy strict pod security rules, or netadmin when you need packet capture.
498. How do you troubleshoot a node in NotReady state?
The short answer
NotReady means the control plane no longer believes the node is healthy: either its kubelet stopped sending heartbeats, or the kubelet itself reports a problem, such as the container runtime being down or the network plugin not ready. Check the node's conditions and events, then the kubelet and the container runtime on the machine itself, its disk, memory and process count, its network path to the API server, and its certificates. On managed clusters, the provider can often repair or replace the node automatically.
bash
kubectl get nodes
kubectl describe node aks-general-12345678-vmss000003Conditions:
Type Status Reason Message
MemoryPressure False KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure True KubeletHasDiskPressure kubelet has disk pressure
PIDPressure False KubeletHasSufficientPID kubelet has sufficient PID available
Ready False KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReadyWhat NotReady means for pods. The node gets taints such as node.kubernetes.io/not-ready or node.kubernetes.io/unreachable. New pods are not scheduled there, and existing pods are evicted after their toleration time, five minutes by default, so their controllers recreate them elsewhere.
Then, on the node itself, through SSH, the cloud's console, or kubectl debug node/... if the node is still partly reachable:
bash
systemctl status kubelet
journalctl -u kubelet --since "30 min ago" | tail -50
systemctl status containerd
crictl ps
df -h; free -mThe usual causes:
| Cause | Typical sign |
|---|---|
| The kubelet stopped or keeps crashing | systemctl status kubelet shows failed; its log says why |
| The container runtime is down | "container runtime is down" in the condition |
| The network plugin is not ready | NetworkPluginNotReady; its DaemonSet pod is failing on this node |
| Full disk | DiskPressure; images and logs filled it |
| Memory or process exhaustion | MemoryPressure, PIDPressure, or the node too overloaded to respond |
| No network to the API server | Heartbeats stop; the node shows Unknown conditions |
| Expired kubelet certificates | Errors about certificates in the kubelet log |
| The virtual machine itself is gone or stuck | The cloud console shows it stopped or failing |
On managed Kubernetes, node auto-repair features, in AKS, EKS and GKE, restart or reimage unhealthy nodes automatically. Often the fastest fix is to drain the node, if possible, and let the node pool replace it, then investigate the cause from logs afterwards (question 426).