Skip to content

Interview PrepKubernetes

Kubernetes Architecture ​

309. What is Kubernetes, and when is an orchestrator useful compared with running containers directly? ​

The short answer

Kubernetes is an open-source system that runs containers across a group of machines and keeps them in the state you describe: how many copies, which image, which settings, reachable at which address. It places containers on machines, restarts or replaces them when they fail, scales them, rolls out new versions and connects them to each other. An orchestrator is worth it when you have many services, need them to survive machine failures, and deploy often. For one small app on one machine, Docker Compose or a managed container service is simpler.

Where it came from. Google ran its own services on an internal system called Borg for over a decade. In 2014 it released Kubernetes, a new open-source system based on those lessons. It is now run by the Cloud Native Computing Foundation, a neutral group under the Linux Foundation, and every major cloud offers it as a managed service. The name is Greek for "helmsman", and it is often written K8s: K, then 8 letters, then s.

The core idea is desired state. You do not tell Kubernetes "start a container on machine 3". You say: "there should always be three copies of checkout-api version 1.4, each with half a CPU, reachable as checkout-api inside the cluster". Kubernetes then works continuously to make reality match that description, and to keep it matching (question 332).

What running containers directly lacks (question 107):

  • If a machine dies, nothing moves its containers elsewhere.
  • Nothing keeps three copies running, or adds more under load.
  • Nothing rolls out a new version safely, pod by pod, checking health.
  • Containers on different machines need hand-built networking to find each other.

When Kubernetes is worth it:

  1. Many services, or many copies, across several machines.
  2. High availability: surviving machine and zone failures.
  3. Frequent deployments with safe rollouts and rollbacks.
  4. Autoscaling with traffic.
  5. A team, or a managed service, able to operate it.

When it is not: one or two small services, or a small team with no one to run the platform. Also when a simpler managed service can host the apps, such as Azure Container Apps, AWS ECS on Fargate or Google Cloud Run (question 559).

310. What problems does Kubernetes solve, and what additional operational complexity does it introduce? ​

The short answer

It solves placing containers on machines efficiently, self-healing, scaling, safe rollouts and rollbacks, service discovery and load balancing, configuration and secrets, storage attachment, and gives one standard API that a huge ecosystem builds on. The price is real: many moving parts to learn, complex networking and security, upgrades several times a year, more layers to debug, lots of YAML, and the cost of running and staffing the platform itself.

Kubernetes solvesKubernetes adds
Placing containers on machines with enough roomA steep learning curve
Restarting and replacing failed containersMany components to understand and monitor
Keeping a number of copies, and autoscalingNetworking layers: CNI, DNS, Services, policies
Rolling updates and rollbacksSecurity work: RBAC, pod security, network policies
Service discovery and load balancingUpgrades about every four months
Configuration and secretsMore layers when debugging
Attaching storageLarge amounts of YAML
A standard API for tools to build onThe cost of the platform team or managed service

About upgrades. Kubernetes releases a new minor version roughly three times a year, and each is supported for about fourteen months. A cluster that is not upgraded regularly falls out of support, and managed providers eventually force upgrades (question 501).

About debugging. A request can fail at the load balancer, the ingress or gateway, the Service, kube-proxy's rules, the network plugin, DNS, a network policy, the pod's readiness, or the app. Each layer is a skill to learn (question 423).

The honest summary for an interview: Kubernetes moves complexity from each application team into one shared platform. That pays off when there are enough services and teams to share it. It does not pay off for a single app, where the platform's complexity is larger than the app's.

311. What is a Kubernetes cluster? ​

The short answer

A cluster is a group of machines, called nodes, managed as one system. It has two parts. The control plane stores the desired state and makes decisions. The worker nodes run the actual containers, grouped into pods. They are joined by a network that lets every pod reach every other pod, and everything is managed through one API.

Picture a restaurant chain's head office and its kitchens. Head office keeps the records and decides what each kitchen should cook. Kitchens do the cooking and report back. Neither works without the other.

What a cluster contains:

  1. The control plane: the API server, the state store etcd, the scheduler and the controllers (question 313).
  2. Worker nodes: virtual or physical machines, each running the kubelet, a container runtime and network components (question 319).
  3. A pod network: every pod gets its own IP address and can reach every other pod's address directly, across nodes (question 365).
  4. Cluster services: DNS, so pods find each other by name, and usually an ingress or gateway for outside traffic.
  5. Storage integration: plugins that attach disks to pods (question 383).

One API for everything. Every action, from deploying an app to reading a log to scaling a service, is a request to the API server. People use kubectl, pipelines and GitOps tools use the same API, and so do the cluster's own components.

Size in practice: clusters range from a single machine on a laptop, using tools such as kind or minikube, to thousands of nodes. Kubernetes' published tested limits are around 5,000 nodes and 150,000 pods per cluster. Many companies run several clusters rather than one huge one, for isolation between environments, regions or teams.

312. What is the difference between the control plane and worker nodes? ​

The short answer

The control plane is the brain: it holds the desired state, makes decisions such as where each pod runs, and runs the loops that fix differences. Worker nodes are the muscles: they run the pods, and report their status back. The control plane does not run your application containers, and worker nodes do not make cluster-wide decisions.

Control planeWorker node
JobStore state, decide, reconcileRun pods
Main partsAPI server, etcd, scheduler, controllerskubelet, container runtime, kube-proxy, network plugin
Runs your appsNoYes
If it is lostNo new changes, but running pods continue (question 484)Its pods are recreated on other nodes
In managed KubernetesRun by the cloud providerYour machines, which you pay for

In managed services such as AKS, EKS and GKE, the provider runs the control plane for you, usually across several zones, and you never see its machines. You manage the worker nodes, grouped into node pools of identical machines (question 611).

In self-managed clusters, for example built with kubeadm, the control plane runs on dedicated machines. They carry a taint, a marker that keeps ordinary pods off them (question 394), so a busy application cannot slow down the cluster's brain. For high availability, there are usually three control-plane machines (question 483).

313. What are the main Kubernetes control-plane components, and what does each do? ​

The short answer

The API server is the front door: every request goes through it, and it is the only part that talks to etcd. etcd is the database that stores the whole cluster's state. The scheduler chooses a node for each new pod. The controller manager runs the controllers, loops that keep making reality match the desired state. In clusters on a cloud, the cloud controller manager handles cloud-specific work, such as creating load balancers.

kubectlpeople, CI, GitOpscontrol planeAPI serverthe front dooretcdall cluster stateschedulerpicks a nodecontroller managerreconcile loopscloud controllermanageronly linkworker node 1kubelet, container runtimekube-proxy, network pluginworker node 2kubelet, container runtimekube-proxy, network pluginnodes also talk onlyto the API server
Everything talks to the API server. No component calls another directly. They all read and write through the API server, and only the API server touches etcd. That one rule is what keeps the system consistent.
ComponentIts jobMore in
kube-apiserverAccepts, checks and stores every change; serves every readQuestion 314
etcdStores all cluster state, reliably, across several machinesQuestions 315 and 483
kube-schedulerAssigns each new pod to a nodeQuestion 316
kube-controller-managerRuns the built-in controllersQuestion 317
cloud-controller-managerCloud-specific controllers: nodes, routes, load balancersQuestion 318

The design rule that matters most: components never talk to each other directly. The scheduler does not call the kubelet. It writes "this pod goes to node 2" into the API server, and node 2's kubelet, which is watching, sees it. Everything is coordinated through stored state. That makes each piece simple, replaceable and able to restart without losing anything.

314. What does the API server do? ​

The short answer

The API server is the single front door to the cluster. Every request, from people, pipelines, controllers and nodes, goes to it over HTTPS. For each request it checks who is asking, checks whether they are allowed, runs admission checks that may change or reject the request, validates the object, and stores it in etcd. It is the only component that reads and writes etcd. It also lets other components watch for changes, which is how the whole cluster reacts to events.

The path of one request, for example kubectl apply of a Deployment:

  1. Authentication: who are you? The API server checks the client's certificate, token, or OIDC identity from a login provider. An unknown client is rejected with 401.
  2. Authorization: may you do this? Usually RBAC, role-based access control, checks whether this identity may create a Deployment in this namespace. If not, 403 (question 415).
  3. Mutating admission. Plug-ins and webhooks may change the object, for example adding default values or injecting a sidecar (question 486).
  4. Validation. Is the object well formed according to its schema?
  5. Validating admission. Policies may reject it, for example "images must come from our registry".
  6. Storage. The object is written to etcd.
  7. Notification. Every component watching Deployments is told about the change.

Watches are the secret of Kubernetes. Instead of asking "anything new?" every second, controllers open a long-lived watch and the API server pushes each change to them as it happens. That is how a new Deployment leads, within moments, to a new ReplicaSet, new pods, a scheduling decision and containers starting.

Some other facts worth knowing:

  • It is stateless: all state lives in etcd. You can run several API servers behind a load balancer for availability and capacity.
  • Everything is versioned resources, such as /apis/apps/v1/namespaces/shop-prod/deployments/checkout-api (question 323).
  • You can see the raw requests kubectl makes with kubectl get pods -v=8.

315. What is etcd, and what kind of data does Kubernetes store in it? ​

The short answer

etcd is a distributed key-value database: a reliable store of keys and values, kept consistent across several machines by a consensus algorithm called Raft. Kubernetes stores its entire state there: every object, such as pods, Deployments, Services, ConfigMaps, Secrets, nodes and namespaces, plus leases and events. It does not store your application's data, container images or logs. Losing etcd without a backup means losing the cluster's whole configuration.

What a key-value store is. It is like a giant dictionary. Each piece of data is stored under a unique name, the key. Kubernetes uses keys shaped like folder paths:

text
/registry/deployments/shop-prod/checkout-api
/registry/pods/shop-prod/checkout-api-7d9f-abcde
/registry/secrets/shop-prod/db-password

The value under each key is the object, stored in a compact binary format.

What is in it:

Stored in etcdNot stored in etcd
Every Kubernetes object and its statusYour application's database data
ConfigMaps and SecretsContainer images, which live in registries
Node records and leasesContainer logs, which stay on nodes
EventsMetrics

Three properties that matter:

  1. Strongly consistent. Every read sees the latest committed write. That is what lets controllers trust what they read.
  2. Replicated with Raft. A write is only committed once a majority of etcd members have it, which is why etcd clusters have an odd number of members (question 483).
  3. Sensitive to slow disks. Every write waits for the disk to confirm it. Slow disks make the whole cluster slow, which is why etcd needs fast storage.

Two operational must-knows:

  • Secrets are only base64-encoded in etcd by default, which is not encryption. Turn on encryption at rest, ideally with a cloud key service (question 379).
  • Back it up regularly, and practise restoring (question 502). In managed clusters, the provider handles etcd for you.

316. What does the scheduler do when a Pod is created? ​

The short answer

The scheduler watches for pods that have no node yet. For each one, it first filters out nodes that cannot run it, for example nodes without enough free CPU or memory for the pod's requests, nodes the pod's rules exclude, or nodes with taints it does not tolerate. It then scores the remaining nodes on preferences such as spreading pods out. It picks the best node and records that choice in the API server. It does not start anything: the chosen node's kubelet does that.

new podneeds 500m CPUand 1Gi memory① filter: who can?node-1: not enough memorynode-2: fitsnode-3: fitsnode-4: taint not toleratednode-5: fits② score: who best?node-2: 61node-3: 87node-5: 74③ bindpod tonode-3
Filter, then score, then bind. Filtering removes nodes that cannot run the pod at all. Scoring ranks the rest by preferences. Binding writes the decision, and the node takes it from there.

1. Filter: which nodes can run it? Checks include:

  • Resources: the node's free allocatable CPU and memory, after the requests of pods already there, must cover this pod's requests (question 390). Actual current use does not matter, only requests.
  • Node selection rules: nodeSelector and node affinity (question 394).
  • Taints and tolerations: a tainted node only accepts pods that tolerate the taint.
  • Ports: a pod asking for a host port needs a node where that port is free.
  • Volumes: a pod needing a disk that exists only in zone A must go to a node in zone A.

2. Score: which is best? Each remaining node gets points from several scoring plug-ins, such as:

  • Spreading pods of the same app across nodes and zones (question 396).
  • Balancing how full nodes are.
  • Preferring nodes that already have the image, which starts faster.
  • Matching preferred, soft affinity rules.

3. Bind. The scheduler writes the chosen node into the pod's record, its spec.nodeName. The kubelet on that node sees a pod assigned to it and starts it (question 319).

When no node fits, the pod stays Pending, and its events say why:

0/5 nodes are available: 1 Insufficient memory, 1 node(s) had untolerated taint {gpu: true}, 3 node(s) didn't match Pod's node affinity.

That message is the first thing to read when a pod will not schedule (question 408).

317. What does the controller manager do? ​

The short answer

The controller manager is one program that runs dozens of controllers. A controller is a loop that watches one kind of object, compares what is desired with what exists, and acts to close the gap. The Deployment controller creates ReplicaSets, the ReplicaSet controller creates and deletes pods to keep the count right, the node controller notices dead nodes, the Job controller runs jobs to completion, and so on. Together they are what makes Kubernetes self-healing.

Some of the controllers it runs:

ControllerWatchesDoes
DeploymentDeploymentsCreates and scales ReplicaSets for rollouts
ReplicaSetReplicaSets, podsKeeps the right number of pods
Node lifecycleNodesMarks silent nodes unhealthy, evicts their pods
Job and CronJobJobs, CronJobsRuns pods to completion, on schedules
EndpointSliceServices, podsKeeps each Service's list of ready pod addresses
NamespaceNamespacesDeletes everything inside a namespace being deleted
Service accountNamespacesCreates the default service account in each namespace
Garbage collectorOwner referencesDeletes objects whose owner is gone (question 487)
Persistent volumeClaims, volumesBinds storage claims to volumes

Why one program for many controllers? Each controller is logically separate, but running them in one process makes the cluster simpler to deploy and run.

High availability. In a cluster with three control-plane machines, three controller managers run, but only one is active at a time. They use leader election: the one holding a lease object in the API server is the leader, and if it stops renewing the lease, another takes over within seconds. This prevents two controllers fighting over the same objects.

Question 332 explains the reconciliation loop in depth.

318. What is the role of the cloud-controller manager in clusters that use it? ​

The short answer

It holds the controllers that need to talk to a specific cloud provider, kept separate from core Kubernetes so each cloud can maintain its own. It has three main jobs. It creates and updates cloud load balancers for Services of type LoadBalancer. It labels nodes with their cloud details, such as zone and machine size, and removes nodes whose virtual machines were deleted. On some networks, it also sets up routes so pods on different nodes can reach each other.

Its three main controllers:

  1. Service controller. When you create a Service of type: LoadBalancer, it calls the cloud's API to create a real load balancer, and points it at the cluster's nodes. Then it writes the load balancer's public address into the Service's status (question 359).
  2. Node controller. It asks the cloud about each node: which zone and region it is in, what machine type it is, and its addresses, and adds those as labels and fields. If the cloud says a node's virtual machine no longer exists, it deletes the node object, so Kubernetes stops waiting for it.
  3. Route controller. On networks where pods get addresses from a separate range, it programs the cloud network's route tables so traffic for each node's pod range reaches that node.

Why it was split out. Originally, cloud-specific code lived inside Kubernetes itself, so every cloud's changes had to go into the main project. Moving it into separate cloud controller managers lets each provider release its own on its own schedule.

In managed Kubernetes, such as AKS, EKS and GKE, the provider runs it for you. You see its work, for example a public IP address appearing on your LoadBalancer Service, but not the component itself.

319. What does kubelet do on a worker node? ​

The short answer

The kubelet is the agent on every node. It registers the node with the cluster and keeps reporting that the node is alive. It watches for pods assigned to its node, and for each one asks the container runtime to pull the images and start the containers, sets up their volumes and networking, and applies their resource limits. It runs their health probes, restarts failed containers, reports their status to the API server, and evicts pods when the node runs short of memory or disk.

API servercontrol planeworker nodekubeletstarts pods, runs probesreports statuscontainer runtimecontainerd, via the CRInetwork pluginstorage driverkube-proxyService rulespodsyour containerswatch, report
The kubelet runs the node. It takes its orders from the API server, works through the container runtime, network plugin and storage driver, and reports back. kube-proxy is a separate agent that only handles Service traffic rules.

What the kubelet does, in detail:

  1. Registers the node and keeps renewing a small Lease object every few seconds, its heartbeat, plus fuller status updates less often. If the heartbeats stop, the control plane marks the node NotReady (question 498).
  2. Watches for its pods: any pod whose spec.nodeName is this node.
  3. Starts them, through the container runtime, using the CRI (question 321): pull the images, create the pod's sandbox and network namespace, then start the init containers in order, then the main containers.
  4. Sets up volumes through storage drivers, mounting ConfigMaps, Secrets and disks.
  5. Applies limits by placing each container in a cgroup (question 100).
  6. Runs probes: startup, readiness and liveness checks (question 401), and restarts containers whose liveness fails.
  7. Restarts crashed containers according to the pod's restart policy, with growing delays, which shows as CrashLoopBackOff (question 407).
  8. Reports status: each pod's phase, conditions and container states go back to the API server.
  9. Protects the node: when memory or disk runs low, it evicts pods to keep the node alive (question 497).
  10. Runs static pods from files on the node (question 485).

The kubelet is not a container itself in most clusters. It is a normal system service on the node, because it has to work even when the cluster's networking or control plane is having problems.

320. What does kube-proxy do in clusters that use it, and how can networking implementations differ? ​

The short answer

kube-proxy runs on every node and makes Services work. It watches Services and their lists of ready pod addresses, and writes rules into each node's kernel so that traffic sent to a Service's virtual address is redirected to one of the ready pods. In its usual modes it does not carry traffic itself; the kernel does, following its rules. Some network plugins, such as Cilium, replace kube-proxy entirely with their own, faster mechanism built on eBPF.

What it solves. A Service gets a stable virtual IP address, its ClusterIP, such as 10.96.14.20. No machine actually has that address. Something on every node must catch packets sent to it and send them to a real pod. That something is kube-proxy's rules.

How it works, in the default iptables mode:

  1. kube-proxy watches the API server for Services and EndpointSlices, the lists of ready pod addresses behind each Service (question 361).
  2. For each Service, it writes rules into the kernel's packet-filtering system, iptables. In plain words: "a packet to 10.96.14.20:80 should have its destination changed to one of 10.244.1.7:8080, 10.244.2.9:8080 or 10.244.3.4:8080, chosen at random".
  3. When a pod sends a request to the Service address, the kernel applies the rule and the packet goes straight to the chosen pod. kube-proxy is not in the path at all.

The different ways it can be done:

ImplementationHow it redirectsNotes
kube-proxy, iptables modeKernel packet-filter rulesThe long-time default; slows down with many thousands of Services
kube-proxy, ipvs modeThe kernel's built-in load balancerScales better, more balancing options
kube-proxy, nftables modeThe newer kernel packet-filter systemThe modern successor to iptables mode
Cilium and other eBPF pluginsSmall programs loaded into the kernelNo kube-proxy needed; fast; adds visibility

eBPF is a kernel feature that lets small, safe programs run inside the kernel at chosen points, for example when a packet arrives. Network plugins use it to route Service traffic, enforce network policies and record network activity, without long chains of rules.

Why it matters in troubleshooting: if a Service resolves by name but connections time out, the Service rules on that node are a suspect. Check that kube-proxy, or its replacement, is healthy. Check that the Service actually has ready endpoints (question 362).

321. What is the Container Runtime Interface (CRI), and why does Kubernetes use it? ​

The short answer

The CRI is a standard set of API calls between the kubelet and a container runtime, for things like "pull this image", "create this pod's sandbox" and "start this container". Because the kubelet only speaks CRI, any runtime that implements it can be used, such as containerd or CRI-O, without changing Kubernetes. It is what allowed Kubernetes to stop using Docker Engine directly in version 1.24 without breaking anyone's images.

The problem it solved. Early Kubernetes had Docker support built directly into the kubelet. Supporting any other runtime meant changing Kubernetes' own code. The CRI, introduced in 2016, put a clean interface in between.

What the interface covers. It is a gRPC API, a fast way for programs to call each other's functions, with two parts:

ServiceExample calls
Image servicePull an image, list images, remove an image
Runtime serviceCreate a pod sandbox, create a container, start it, stop it, stream its logs, run exec

The runtimes that implement it:

  • containerd, with its built-in CRI plug-in, the most common, used by AKS, EKS, GKE and most others.
  • CRI-O, a runtime made specifically for Kubernetes, common in OpenShift.

Underneath, both use an OCI low-level runtime such as runc to actually create containers (question 33).

The dockershim story. Docker Engine never implemented the CRI. Kubernetes kept a translation layer called dockershim inside the kubelet to talk to it, until Kubernetes 1.24 removed it in 2022. Clusters switched to containerd, which Docker itself uses underneath. Images built with Docker kept working, because they are standard OCI images (question 461). You can still inspect a node's containers with crictl, a CRI-level command-line tool.

322. What is kubectl, and how does it communicate with a cluster? ​

The short answer

kubectl is the command-line tool for Kubernetes. It reads a kubeconfig file to find the cluster's API server address, the certificate to trust, and your credentials. Every command becomes one or more HTTPS requests to the API server's REST API, such as GET /api/v1/namespaces/shop-prod/pods, and the responses are formatted for you. It has no special access: it can only do what your identity is authorised to do.

The kubeconfig file, by default ~/.kube/config, has three kinds of entries:

yaml
clusters:
  - name: prod
    cluster:
      server: https://prod-aks-1a2b.hcp.westeurope.azmk8s.io:443   # (1)
      certificate-authority-data: LS0tLS1CRUdJTi...                 # (2)
users:
  - name: asha@shop
    user:
      exec:                                                         # (3)
        command: kubelogin
        args: [get-token, --login, azurecli, --server-id, 6dae42f8-4368-4678-94ff-3960e28e3630]
contexts:
  - name: prod
    context: { cluster: prod, user: asha@shop, namespace: shop-prod }   # (4)
current-context: prod
  1. The API server's address.
  2. The certificate authority that signed the API server's certificate, so kubectl can be sure it is talking to the real cluster.
  3. How to get credentials. Here, a helper program fetches a short-lived token from Microsoft Entra ID, Azure's identity service, using your own login.
  4. A context joins a cluster, a user and a default namespace under one name (question 328).

Seeing the requests:

bash
kubectl get pods -n shop-prod -v=6
GET https://prod-aks-1a2b.hcp.westeurope.azmk8s.io:443/api/v1/namespaces/shop-prod/pods?limit=500 200 OK in 87 milliseconds

-v=6 shows each request and its result. Higher levels, up to 9, show headers and full bodies.

Useful habits:

  • kubectl explain deployment.spec.strategy describes any field, straight from the cluster's own schema.
  • -o yaml, -o json and -o jsonpath=... give machine-readable output for scripts.
  • Plug-ins: any program named kubectl-something on your PATH becomes kubectl something. krew is a plug-in manager with hundreds of them.

483. Why does etcd need an odd number of members, and what is quorum? ​

The short answer

etcd only accepts a write once a majority of its members have stored it. That majority is called the quorum: for n members, more than half, which is n divided by 2, rounded down, plus 1. A cluster keeps working as long as a quorum is alive. Three members tolerate one failure and five tolerate two. Four members also tolerate only one, because their quorum is three, so the fourth member adds cost and risk without adding resilience. That is why etcd clusters have odd sizes.

Why a majority? etcd uses the Raft consensus algorithm. One member is elected leader, and every write goes through it. The leader sends the write to the others and commits it only when a majority has it. Requiring a majority means two separate groups can never both accept writes, because two different majorities of the same cluster must share at least one member, who would notice the conflict.

3 members, 1 down2 alive, quorum is 2still works4 members, 2 down2 alive, quorum is 3stops accepting writes5 members, 2 down3 alive, quorum is 3still worksGreen: running member. Red: failed member.
Quorum in practice. The cluster needs more than half of all members, counting the failed ones too. Adding a fourth member raises the quorum without letting the cluster survive any more failures.
MembersQuorumFailures it survives
110
220
321
431
532
743

What losing quorum means. The remaining members refuse writes, so the API server cannot save any change. Kubernetes cannot schedule, scale or update anything until quorum returns. Reads may also stop, because etcd's normal reads also go through the quorum to stay consistent.

Why not just add many members? Every write must reach a majority, so more members make each write slower. Three is the usual choice, five for very important clusters, and members are spread across availability zones so one zone failing never removes a majority.

Network splits. If a network failure splits five members into groups of three and two, the group of three still has quorum and keeps working. The group of two cannot write. There is never a moment when both sides accept conflicting changes.

484. What happens if the Kubernetes control plane goes down; do running applications stop? ​

The short answer

No, running applications keep running. Each kubelet keeps its existing containers alive and even restarts crashed ones on the same node, and Service traffic keeps flowing through the rules already written on each node. But the cluster can no longer change: nothing new is scheduled, nothing scales, no deployments happen, kubectl stops working, and if a node dies, its pods are not replaced. Endpoint lists also stop updating, so traffic can keep going to pods that have died.

Still worksStops working
Existing containers keep runningkubectl and every API request
Crashed containers restart on the same nodeScheduling new pods
Service traffic, using existing node rulesDeployments, rollouts and rollbacks
DNS for existing Services, served from CoreDNS's cacheAutoscaling, both pods and nodes
Already-mounted volumesReplacing pods from a failed node
Updating which pods are behind a Service
CronJobs firing, Jobs being created

Why apps keep running. The kubelet and the container runtime run on each node independently. They already know which containers should be there, and they do not need the control plane to keep them running.

Why it still gets worse over time:

  1. No self-healing across nodes. If a node fails, its pods are gone, and nothing creates replacements.
  2. Stale endpoints. If a pod dies, the Service's endpoint list is not updated, so some requests keep going to a dead address and fail.
  3. No reaction to load. Traffic grows, and nothing scales.
  4. No fixes. You cannot deploy anything, including the fix for whatever caused the incident.

How clusters protect against it:

  • Highly available control planes: several API servers behind a load balancer, three or five etcd members across zones (question 483).
  • Managed Kubernetes: the provider runs and repairs the control plane, often with a published uptime commitment.
  • Backups of etcd, and practised restores (question 502).

485. What is a static pod? ​

The short answer

A static pod is a pod defined by a manifest file on a node, in a folder the kubelet watches, such as /etc/kubernetes/manifests. The kubelet runs it directly, without the API server or the scheduler deciding anything. A read-only "mirror" of it appears in the API so you can see it, but you cannot really manage it through kubectl: delete it, and the kubelet starts it again from the file. Its main use is running the control plane itself in clusters built with tools such as kubeadm.

The chicken-and-egg problem it solves. The API server runs as a pod. But pods are normally created through the API server. How does the first API server start? In a kubeadm cluster, the answer is static pods: the kubelet on each control-plane machine reads these files and starts the control plane from them.

bash
ls /etc/kubernetes/manifests/
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

How static pods behave:

  1. Started by the kubelet as soon as a file appears in the folder, and restarted if they crash.
  2. Changed by editing the file. The kubelet notices the change and recreates the pod. This is how kubeadm upgrades control-plane components.
  3. Removed by deleting the file.
  4. Visible as mirror pods, named after the node, such as kube-apiserver-cp-1. Deleting the mirror pod through the API has no lasting effect.
  5. Not managed by controllers: no ReplicaSet, no Deployment, no scheduling.

In managed Kubernetes, you usually never see them, because the provider runs the control plane outside your nodes. They still matter in interviews because they explain how a cluster starts itself, and why "I deleted it and it came back" can happen.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet