Skip to content

Interview PrepKubernetes

Resources, Scheduling and Scaling ​

389. What is the difference between resource requests and resource limits for CPU and memory? ​

The short answer

A request is the amount of CPU or memory a container is guaranteed, and the scheduler reserves it on a node when placing the pod. A limit is the most the container is allowed to use. Above its CPU limit, a container is slowed down, which is called throttling. Above its memory limit, it is killed. Requests decide where pods go and how CPU is shared under pressure. Limits decide what happens when a container tries to use more.

yaml
resources:
  requests:
    cpu: 250m          # (1)
    memory: 384Mi      # (2)
  limits:
    memory: 512Mi      # (3)
  1. A quarter of a CPU core. m means thousandths, so 1000m is one full core.
  2. 384 mebibytes. Mi and Gi are powers of two: 1 MiB is 1,048,576 bytes.
  3. The memory ceiling. Above 512 MiB, the kernel kills the container. There is no CPU limit here, which is a common and deliberate choice, explained below.
RequestLimit
MeaningGuaranteed minimumHard maximum
Used by the schedulerYesNo
CPU above itAllowed, if the node has spareThrottled
Memory above itAllowed, but evicted first under pressureKilled (question 391)
Sets the QoS classYes, together with limits (question 393)Yes

Sensible defaults for a web service:

  1. Memory request at the app's normal working size, and memory limit equal to it or slightly above. Memory cannot be taken back gently, so a predictable ceiling is safer.
  2. CPU request at the app's typical use, so the scheduler places pods realistically.
  3. CPU limit: many teams leave it unset. Without it, a container can use idle CPU on the node, and is only held back, by its request's weight, when others need CPU too. With it, the container is throttled even when the node is idle (question 392). Some environments still require CPU limits, for example for cost control or strict fairness between tenants.

390. How does the scheduler use CPU and memory requests when placing a Pod? ​

The short answer

For each node, the scheduler adds up the requests of all the pods already placed there. It compares that total with the node's allocatable capacity. Allocatable capacity is the node's real capacity, minus what is set aside for the operating system and Kubernetes itself. A new pod fits only if its own requests fit in what is left. Actual current usage plays no part: a node can be nearly idle and still count as full if its pods requested a lot.

A worked example. A node has 4 CPUs and 16 GiB of memory.

CPUMemory
Node capacity4000m16 GiB
Reserved for the system and Kubernetes180m1.5 GiB
Allocatable3820m14.5 GiB
Requests of pods already here3400m9 GiB
Free for scheduling420m5.5 GiB

A new pod requesting 500m CPU and 1Gi memory does not fit: it needs 500m, only 420m is free. It does not matter that the existing pods are actually using only 1,200m right now.

bash
kubectl describe node aks-general-12345678-vmss000003 | grep -A 8 "Allocated resources"
Allocated resources:
  Resource           Requests      Limits
  cpu                3400m (89%)   5200m (136%)
  memory             9Gi (62%)     12Gi (82%)

Limits can add up to more than 100%, which is called overcommitment: it works as long as not every pod uses its full limit at once.

Why requests, not usage? Usage changes every second. Scheduling on usage would pack pods onto a node during a quiet minute, and they would fight for resources at the next busy one. Requests give stable, predictable placement.

The consequences:

  • Requests that are too high waste money: nodes look full while idle.
  • Requests that are too low, or missing, overpack nodes: pods get evicted or slowed when everything gets busy. A pod with no requests counts as needing nothing, so the scheduler can put any number of them on one node.

391. What happens when a container exceeds its memory limit? ​

The short answer

The kernel kills it, because the limit is enforced by the container's cgroup. Kubernetes records the reason as OOMKilled, with exit code 137, and restarts the container according to the pod's restart policy, with the same limit. If it keeps happening, the pod goes into CrashLoopBackOff. This is different from an eviction, where the kubelet removes a pod because the whole node is short of memory.

What you see:

bash
kubectl get pod checkout-api-7d9f-abcde -n shop-prod
kubectl describe pod checkout-api-7d9f-abcde -n shop-prod | grep -A 6 "Last State"
NAME                      READY   STATUS             RESTARTS      AGE
checkout-api-7d9f-abcde   0/1     CrashLoopBackOff   5 (40s ago)   12m

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 27 Oct 2026 10:41:02 +0000
      Finished:     Mon, 27 Oct 2026 10:43:47 +0000

What happened: the container's memory use hit its limit, the kernel's out-of-memory killer ended its process with SIGKILL, and the kubelet restarted it. The same mechanism as in plain Docker (question 97).

OOM kill versus eviction:

OOM killEviction
Caused byThe container reaching its own limitThe node running short of memory
Done byThe kernelThe kubelet
What happensThe container restarts, same podThe pod is removed; its controller makes a new one, maybe elsewhere
Shows asOOMKilled, exit code 137Status Failed, reason Evicted (question 497)

What to do is question 411: find out whether the app needs more memory, has a leak, or is sizing its own memory wrongly, before changing the limit.

392. What is CPU throttling, and how can it affect application latency? ​

The short answer

A CPU limit is enforced in short periods, 100 milliseconds by default: in each period, the container may use its limit's worth of CPU time, across all its threads. A multi-threaded app can use that allowance in the first part of the period, and is then frozen until the next period starts. That pause is throttling. Average CPU use can look low, yet requests that arrive during a frozen stretch wait, sometimes for tens of milliseconds, which shows up as slow responses at the high percentiles.

0 ms25 ms100 ms4 threads runningthrottled: frozen until the next perioda request arriving now waits up to 70 msLimit 1 CPU = 100 ms of CPU time per 100 ms. Four threads spend it in 25 ms.
How a CPU limit freezes a busy container. The limit is a budget per period, shared by all threads. Spent early, the rest of the period is a pause, whatever the average CPU use looks like.

The arithmetic. A limit of 1 CPU means 100 ms of CPU time per 100 ms period. If four threads are busy at once, they consume 4 ms of budget every millisecond, so the budget is gone after 25 ms. The container is then frozen for the remaining 75 ms.

Why it surprises people. Over a whole minute, the container may average only 40% of its limit, so dashboards look healthy. But its work comes in bursts, such as a burst of requests, or garbage collection running on several threads, and each burst runs into the wall.

How to see it. The cgroup counts throttled periods (question 157). In Prometheus, the cAdvisor metrics container_cpu_cfs_throttled_periods_total divided by container_cpu_cfs_periods_total give the share of periods in which the container was throttled. More than a few percent on a latency-sensitive service is worth fixing.

How to reduce it:

  1. Remove the CPU limit, keeping a realistic CPU request. The request still protects other pods' fair share when the node is busy (question 389).
  2. Raise the limit, if your environment requires limits.
  3. Match the runtime's threads to the limit. For example, Go's GOMAXPROCS, the number of threads running Go code at once, can be set to the container's CPU limit rather than the node's core count. Modern Java and newer Go versions read the container's limit automatically.

393. What are QoS classes such as Guaranteed, Burstable, and BestEffort, and how are they assigned? ​

The short answer

Kubernetes gives every pod a quality-of-service class based on its requests and limits. Guaranteed means every container has CPU and memory requests equal to its limits. BestEffort means no container has any requests or limits. Burstable is everything in between. The class decides who is removed first when a node runs out of memory: BestEffort pods first, then Burstable pods using more than they requested, and Guaranteed pods last.

ClassRuleUnder node memory pressure
GuaranteedEvery container: CPU and memory requests equal limitsEvicted last
BurstableSome requests or limits, but not GuaranteedEvicted when using more than requested
BestEffortNo requests or limits at allEvicted first
yaml
# Guaranteed
resources:
  requests: { cpu: "1", memory: 1Gi }
  limits:   { cpu: "1", memory: 1Gi }
---
# Burstable
resources:
  requests: { cpu: 250m, memory: 384Mi }
  limits:   { memory: 512Mi }
---
# BestEffort: no resources section at all
bash
kubectl get pod checkout-api-7d9f-abcde -o jsonpath='{.status.qosClass}'
Burstable

Where the class is used:

  1. Eviction order when the node is short of memory or disk (question 497).
  2. The kernel's OOM priority. Kubernetes sets an OOM score adjustment for each container: BestEffort containers are the kernel's first choice to kill if the whole node runs out of memory, and Guaranteed containers the last.

Practical advice: important services should be Burstable with realistic requests, or Guaranteed for the most critical ones. Avoid BestEffort for anything that matters: it is the first to go, and it lets the scheduler overpack nodes.

394. How do taints, tolerations, node selectors, and node affinity influence Pod placement? ​

The short answer

Taints are set on nodes and repel pods: a tainted node only accepts pods that tolerate the taint. Tolerations are set on pods and allow them onto tainted nodes, but do not pull them there. Node selectors and node affinity are set on pods and attract them to nodes with certain labels, either as a hard rule or as a preference. To dedicate nodes to one kind of workload, combine both: a taint keeps others off, and affinity draws the right pods on.

Taints and tolerations: "keep out, unless".

bash
kubectl taint nodes gpu-node-1 gpu=true:NoSchedule
yaml
tolerations:
  - key: gpu
    operator: Equal
    value: "true"
    effect: NoSchedule
Taint effectMeaning
NoScheduleNew pods without the toleration are not placed here
PreferNoScheduleAvoid placing them here if possible
NoExecuteAlso evicts running pods without the toleration, optionally after tolerationSeconds

Kubernetes adds taints itself too. For example, a node that stops responding gets node.kubernetes.io/unreachable:NoExecute, which evicts its pods after a few minutes, so they are recreated elsewhere.

Node selectors and affinity: "go there".

yaml
nodeSelector:
  node.kubernetes.io/instance-type: Standard_NC6s_v3          # (1)
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:          # (2)
      nodeSelectorTerms:
        - matchExpressions:
            - { key: kubernetes.io/arch, operator: In, values: [amd64] }
    preferredDuringSchedulingIgnoredDuringExecution:         # (3)
      - weight: 50
        preference:
          matchExpressions:
            - { key: topology.kubernetes.io/zone, operator: In, values: [westeurope-1] }
  1. A simple hard rule: only nodes with this label.
  2. Node affinity's hard form, with more operators, such as In, NotIn, Exists, Gt and Lt.
  3. A soft preference: zone 1 earns extra points in scoring, but other zones are allowed. "IgnoredDuringExecution" means a running pod is not moved if node labels change later.

Dedicated GPU nodes, the classic combination: taint the GPU nodes so ordinary pods stay off, and give GPU workloads both the toleration and an affinity for GPU nodes, so they go there and nowhere else.

395. What are pod affinity and anti-affinity, and when would you use them? ​

The short answer

Pod affinity places a pod near other pods with certain labels, and pod anti-affinity places it away from them, where "near" means within the same topology domain, such as the same node or the same zone. Use anti-affinity to keep copies of one service on different nodes or zones, so one failure does not take them all. Use affinity to put a pod close to something it talks to a lot, such as a cache, to cut network latency.

yaml
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels: { app.kubernetes.io/name: checkout-api }
        topologyKey: kubernetes.io/hostname                     # (1)
  podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels: { app.kubernetes.io/name: checkout-cache }
          topologyKey: topology.kubernetes.io/zone              # (2)
  1. Hard anti-affinity per node: never place two checkout-api pods on the same node. If you ask for five replicas and have four nodes, the fifth stays Pending.
  2. Soft affinity per zone: prefer zones where a cache pod already runs.

The topologyKey is a node label that defines the domain: kubernetes.io/hostname means "the same node", and topology.kubernetes.io/zone means "the same availability zone".

Hard rules have sharp edges. Required anti-affinity can block scaling and rollouts when there are not enough domains, and it is relatively expensive for the scheduler to evaluate in large clusters. For simply spreading replicas evenly, topology spread constraints are usually the better tool (question 396).

396. What is a topology spread constraint, and how can it improve fault distribution? ​

The short answer

A topology spread constraint tells the scheduler to keep pods of a group evenly spread across domains, such as zones or nodes, within a tolerance called the maximum skew. With six replicas and three zones and a maximum skew of 1, you get two per zone, so losing one zone loses only a third of the capacity. You can make the rule strict, refusing to schedule rather than break it, or soft, preferring balance but allowing imbalance.

zone 1zone 2: failszone 3maxSkew 1 across zones: 2, 2, 2One zone lost still leaves 4 of 6 pods serving.
Spreading across failure domains. Each zone is a separate data centre. Keeping replicas balanced means no single zone failure can take most of the service down.
yaml
topologySpreadConstraints:
  - maxSkew: 1                                          # (1)
    topologyKey: topology.kubernetes.io/zone            # (2)
    whenUnsatisfiable: DoNotSchedule                    # (3)
    labelSelector:
      matchLabels: { app.kubernetes.io/name: checkout-api }
    matchLabelKeys: [pod-template-hash]                 # (4)
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway                   # (5)
    labelSelector:
      matchLabels: { app.kubernetes.io/name: checkout-api }
  1. The most allowed difference between the busiest and the emptiest domain. With 1, counts such as 2, 2, 2 or 3, 2, 2 are fine, and 3, 3, 0 is not.
  2. Balance across availability zones.
  3. Strict: if placing a pod would break the rule, leave it Pending.
  4. Count only pods from the same ReplicaSet. During a rollout, old and new pods are then balanced separately, which avoids lopsided placement once the old ones are gone.
  5. A second, softer rule: also spread across nodes when possible.

Why this beats anti-affinity for spreading: it allows more than one pod per domain while keeping them balanced, so it keeps working when replicas outnumber nodes or zones.

397. What is a Horizontal Pod Autoscaler (HPA), and which metrics can drive it? ​

The short answer

The Horizontal Pod Autoscaler changes the number of replicas of a Deployment or StatefulSet to keep a metric near a target. It can use three kinds of metric. Resource metrics, such as average CPU use as a percentage of the pods' requests. Custom metrics from the pods themselves, such as requests per second. Or external metrics, such as the length of a message queue. Its basic formula is: desired replicas equal current replicas times the current metric divided by the target, rounded up.

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
  namespace: shop-prod
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout-api }
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 60 }       # (1)
    - type: Pods
      pods:
        metric: { name: http_requests_per_second }
        target: { type: AverageValue, averageValue: "40" }          # (2)
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300                               # (3)
      policies: [{ type: Percent, value: 50, periodSeconds: 60 }]
  1. Keep average CPU at 60% of the pods' CPU requests.
  2. Also keep requests per second per pod at about 40. With several metrics, the HPA calculates a replica count for each and uses the largest.
  3. Scale down slowly. Act only on the lowest recommendation of the last five minutes, which is the default. Remove at most half the pods per minute. Then a short dip in traffic does not cause a big scale-down.

A worked example. Four replicas are running at 90% average CPU, against a target of 60%:

text
desired = ceil(4 × 90 / 60) = ceil(6) = 6 replicas

The HPA ignores small differences: by default, if the ratio is within 10% of the target, it does nothing, which prevents constant small changes.

Where the metrics come from:

KindSourceExample
Resourcemetrics-serverCPU, memory
Pods and Object, customAn adapter, such as the Prometheus adapterRequests per second
ExternalAn adapter, or KEDA (question 496)Queue length

metrics-server is the small cluster add-on that collects CPU and memory use from each kubelet. It must be installed for CPU-based autoscaling, and for kubectl top, to work (question 635).

398. Why might HPA fail to scale a Deployment even when CPU usage is high? ​

The short answer

The most common reason is that the pods have no CPU request, so utilisation cannot be calculated and the HPA shows the current value as unknown. There are other reasons too. metrics-server may be missing or broken. The HPA may already be at its maximum replicas, or target the wrong workload. Its stabilisation rules may be holding it back. Or it did scale, but the new pods are stuck Pending, because the cluster has no room. kubectl describe hpa usually says which.

bash
kubectl get hpa checkout-api -n shop-prod
kubectl describe hpa checkout-api -n shop-prod
NAME           REFERENCE                 TARGETS         MINPODS   MAXPODS   REPLICAS
checkout-api   Deployment/checkout-api   <unknown>/60%   3         10        3

Conditions:
  Type           Status  Reason                   Message
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container "api"

The usual causes, and what shows:

CauseWhat you see
No CPU request on a container<unknown> target; "missing request for cpu"
metrics-server missing or failing<unknown>; kubectl top pods fails too
Already at maxReplicasCondition ScalingLimited with "the desired replica count is more than the maximum"
Wrong scaleTargetRefEvents say the target cannot be found
Waiting on stabilisationIt scales, but later than expected
Scaled, but no roomReplicas went up; new pods Pending (question 408)
High CPU in only one podThe average across pods is still below target
Pods not ready yetUnready pods' metrics are treated cautiously during scale-up

A detail about multi-container pods. CPU utilisation is calculated against the requests of all containers in the pod together. A sidecar with a large CPU request and little use can hide a busy app container. The ContainerResource metric type targets one container's CPU only.

And sometimes the HPA is right not to scale. Sometimes the work is limited by something other than CPU, such as a database or a lock. Then more pods do not help, and the right metric to scale on is a different one.

399. How do Pod autoscaling and cluster/node autoscaling differ? ​

The short answer

Pod autoscaling changes how many pods run, with the HPA, or how big each pod's requests are, with the VPA, inside the room the cluster already has. Node autoscaling changes how much room there is: it adds nodes when pods cannot be scheduled for lack of resources, and removes nodes that are underused. They work as a chain: the HPA adds pods, some cannot fit and stay Pending, the node autoscaler sees them and adds nodes, and the pods start there.

traffic risesCPU at 90%HPA4 to 8 replicas3 pods Pendingnodes are fullnode autoscaleradds a nodepods runon new nodePod scaling takes seconds. Adding a node takes minutes.Keep some headroom, or scale earlier, to cover the gap.
Two autoscalers in a chain. The pod autoscaler asks for more copies. The node autoscaler makes room for them. Neither knows about traffic directly beyond its own signal.
Pod autoscalingNode autoscaling
ChangesNumber of pods, or their sizeNumber of nodes
ToolsHPA, VPA, KEDACluster Autoscaler, Karpenter, cloud node auto-provisioning
Triggered byMetrics such as CPU or queue lengthPods Pending for lack of resources; underused nodes
SpeedSecondsMinutes, while a machine starts

Two node autoscalers you will hear about:

  1. Cluster Autoscaler: works with node groups, such as AKS node pools or EKS node groups, each with a fixed machine type. It grows the group that can fit the pending pods, and shrinks groups whose nodes are underused and whose pods can move elsewhere.
  2. Karpenter: started at AWS, now also used for Azure through its node auto-provisioning feature. It looks at the pending pods' actual needs, such as CPU, memory, architecture and zone, and creates a suitably sized machine directly, often choosing cheaper or better-fitting types. It also consolidates, replacing several half-empty nodes with fewer ones.

Scale-down needs care. Removing a node means evicting its pods. The node autoscaler respects PodDisruptionBudgets (question 400), and pods with local storage or special annotations can prevent a node from being removed.

The timing gap. A traffic spike can outrun node provisioning. Common answers are a little spare capacity, often provided by low-priority placeholder pods that are evicted to make room, and scaling on a leading signal, such as queue length, rather than CPU.

400. What is a PodDisruptionBudget, and what does it protect during voluntary disruptions? ​

The short answer

A PodDisruptionBudget limits how many pods of a group can be taken down at the same time by voluntary actions, such as draining a node for maintenance, a node autoscaler removing a node, or a cluster upgrade. You set either the minimum number that must stay available or the maximum that may be unavailable. Evictions that would break the budget are refused and retried later. It does not protect against involuntary failures, such as a node crashing.

yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
  namespace: shop-prod
spec:
  maxUnavailable: 1                                   # (1)
  selector:
    matchLabels: { app.kubernetes.io/name: checkout-api }
  unhealthyPodEvictionPolicy: AlwaysAllow             # (2)
  1. At most one checkout-api pod may be evicted at a time. minAvailable: 2 would express a similar rule from the other side.
  2. Pods that are already unhealthy may always be evicted, so a broken pod cannot block maintenance.

Voluntary versus involuntary:

Voluntary, PDB appliesInvoluntary, PDB does not apply
kubectl drain for node maintenanceNode hardware failure
Node autoscaler scale-downKernel panic, VM deleted
Managed cluster upgradesOut-of-memory kill
Evictions through the Eviction APINetwork partition

PDBs work through the Eviction API: tools that ask to evict a pod politely are told "not now" if the budget would be broken. Deleting a pod directly with kubectl delete pod bypasses the budget.

Budgets that block everything:

  • minAvailable equal to the replica count, or maxUnavailable: 0, means no voluntary eviction can ever happen, so node drains and upgrades hang forever (question 426).
  • A single-replica app with minAvailable: 1 has the same effect. Either run two replicas, or accept the brief disruption.

PDBs and rollouts are separate. A Deployment's rolling update is controlled by maxUnavailable in its own strategy, not by the PDB.

493. What are ResourceQuotas and LimitRanges? ​

The short answer

A ResourceQuota caps what a whole namespace may use in total: for example, at most 20 CPUs of requests, 64 GiB of memory limits, 100 pods and 2 load balancers. A LimitRange sets rules for each individual container or pod in a namespace: default requests and limits for containers that do not set their own, and minimum and maximum values. Together they stop one team or one mistake from consuming a shared cluster, and make sure every pod gets sensible resource settings.

yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: shop-dev-quota
  namespace: shop-dev
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 48Gi
    limits.memory: 64Gi
    pods: "100"
    persistentvolumeclaims: "20"
    services.loadbalancers: "2"           # (1)
---
apiVersion: v1
kind: LimitRange
metadata:
  name: shop-dev-defaults
  namespace: shop-dev
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }   # (2)
      default:        { memory: 256Mi }              # (3)
      max:            { memory: 2Gi }                # (4)
  1. Load balancers cost money, so a quota on them is common.
  2. Containers with no requests get these, added at admission.
  3. Containers with no limit get a 256 MiB memory limit.
  4. No single container may ask for more than 2 GiB.

How they work together. When a namespace has a quota on CPU or memory, every new pod must declare those resources, or it is rejected. A LimitRange's defaults fill them in automatically, so teams are not blocked by forgetting.

When the quota is full, new pods are rejected at admission, not left pending:

Error creating: pods "checkout-api-5c8b-x2k9" is forbidden: exceeded quota: shop-dev-quota, requested: requests.cpu=500m, used: requests.cpu=19800m, limited: requests.cpu=20

That message often appears only in the ReplicaSet's events, not on any pod, because the pod was never created (question 428).

494. What are PriorityClasses and preemption? ​

The short answer

A PriorityClass gives pods a priority number, and the higher the number, the more important the pod. When a high-priority pod cannot be scheduled because the cluster is full, the scheduler may preempt lower-priority pods: evict them to make room. Priority also affects which pods are evicted first when a node runs short of resources. Use it to protect critical services over batch jobs, carefully, because misused priorities let one workload push others out.

yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: shop-critical
value: 100000                              # (1)
description: Customer-facing checkout path
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-low
value: 1000
preemptionPolicy: Never                    # (2)
---
# in a pod spec:
priorityClassName: shop-critical
  1. A bigger number means a more important pod. Kubernetes reserves very large values, around two billion, for its own system-cluster-critical and system-node-critical classes, used by components such as DNS.
  2. Pods of this class never evict others to get scheduled. They wait instead.

How preemption works:

  1. A shop-critical pod is Pending: no node has room.
  2. The scheduler looks for a node where evicting some lower-priority pods would make enough room.
  3. It evicts them, giving them their normal graceful shutdown, and records which node it chose for the waiting pod.
  4. The critical pod is scheduled there once the room is free.

Uses:

  • Checkout and payment services above internal tools and batch jobs.
  • Placeholder pods with negative or very low priority, which hold spare capacity and are instantly preempted when real pods need room (question 399).

Risks: if every team marks its pods as highest priority, priority means nothing. And preemption ignores PodDisruptionBudgets when it has no other choice, so treat it as a last resort, not normal scheduling.

495. What is the Vertical Pod Autoscaler, and can it run together with the Horizontal Pod Autoscaler? ​

The short answer

The Vertical Pod Autoscaler watches how much CPU and memory pods really use over time, and recommends, or sets, better requests for them. In its recommendation-only mode, it simply tells you good values. In its active modes, it applies them, traditionally by recreating pods. It should not manage the same CPU or memory metric that an HPA is scaling on, because the two would fight. VPA on requests plus an HPA on a different metric, such as requests per second, works well.

yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api
  namespace: shop-prod
spec:
  targetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout-api }
  updatePolicy:
    updateMode: "Off"                       # (1)
  resourcePolicy:
    containerPolicies:
      - containerName: api
        minAllowed: { cpu: 100m, memory: 256Mi }
        maxAllowed: { cpu: "2", memory: 2Gi }
  1. Recommendation only. kubectl describe vpa checkout-api then shows a recommended request for each container, based on its real usage history.
Update modeBehaviour
OffRecommends only
InitialSets requests when pods are created, never changes them after
RecreateEvicts pods whose requests are far from the recommendation, so they come back with new values
InPlaceOrRecreateUses in-place resizing where the cluster supports it, otherwise recreates

In-place resizing. Newer Kubernetes versions can change a running pod's CPU and memory without restarting it. VPA can use that, which removes most of the disruption of its active modes.

Why VPA and HPA conflict on the same metric. Suppose the HPA keeps CPU at 60% of requests, and VPA raises the requests because usage is high. Utilisation, being usage divided by requests, drops, so the HPA removes pods, so usage per pod rises, so VPA raises requests again. The two keep undoing each other.

A safe combination: VPA in recommendation mode to set sensible requests during reviews, plus an HPA scaling on CPU or on a custom metric.

496. What is KEDA, and when would you scale on something like queue length? ​

The short answer

KEDA, Kubernetes Event-driven Autoscaling, scales workloads based on outside event sources, such as the number of messages waiting in a queue, the lag of a Kafka consumer group, a Prometheus query or a schedule. It can scale a workload all the way to zero when there is nothing to do, and back up when work arrives. Scale on queue length when the queue, not CPU, is the true measure of how much work is waiting, which is the case for most background workers.

The shop's notification worker reads "order placed" events from an Azure Service Bus queue. Its CPU is low even when thousands of emails are waiting, because each email mostly waits on the email provider. CPU-based scaling would never add workers. The queue length tells the real story.

yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: notification-worker
  namespace: shop-prod
spec:
  scaleTargetRef: { name: notification-worker }        # (1)
  minReplicaCount: 0                                    # (2)
  maxReplicaCount: 30
  triggers:
    - type: azure-servicebus
      metadata:
        queueName: order-placed
        namespace: shop-bus
        messageCount: "50"                              # (3)
      authenticationRef: { name: servicebus-workload-identity }   # (4)
  1. The Deployment to scale.
  2. Scale to zero when the queue is empty, which saves money for workloads that are idle much of the time.
  3. Aim for about 50 waiting messages per replica. With 1,000 messages waiting, KEDA asks for 20 replicas.
  4. KEDA reads the queue's length using workload identity, without stored passwords.

How it works. KEDA runs a small operator and a metrics adapter. For each ScaledObject, it creates and manages a normal HPA fed with the external metric, and it handles the special step between zero and one replica itself, because a normal HPA cannot scale to zero.

Good fits: queue workers, stream consumers, scheduled bursts, and anything whose demand is measured outside the pods. For run-to-completion work, KEDA's ScaledJob creates Jobs instead of scaling a Deployment.

497. What are node-pressure evictions, and why do pods show up as Evicted? ​

The short answer

When a node runs low on memory, disk space or process IDs, the kubelet evicts pods to protect the node itself, choosing first those using the most above their requests, and lower-priority, lower-QoS pods before others. An evicted pod shows status Failed with reason Evicted and a message saying which resource ran low, and its controller creates a replacement, perhaps on another node. Containers writing too much to their local disk can also be evicted for exceeding their ephemeral-storage limit.

What the kubelet watches, with typical default thresholds:

SignalTypical hard threshold
memory.availableBelow 100 MiB
nodefs.available, the node's main diskBelow 10%
imagefs.available, the disk holding imagesBelow 15%
pid.availableClose to the node's process limit

Clusters can also set soft thresholds with grace periods, so short spikes do not trigger evictions. Managed services set their own values.

Who goes first:

  1. Pods using more than their request of the scarce resource.
  2. Among those, lower priority first (question 494).
  3. Then by how far above their request they are.

In practice, BestEffort pods go first and Guaranteed pods last (question 393).

What it looks like:

bash
kubectl get pods -n shop-prod --field-selector=status.phase=Failed
kubectl describe pod report-builder-6d4f-pq2xz -n shop-prod | grep -A 2 Reason
Reason:   Evicted
Message:  The node was low on resource: memory. Threshold quantity: 100Mi, available: 87Mi. Container builder was using 2804Mi, request is 512Mi, has larger consumption of memory.

The message names the guilty container and how far it was above its request.

Disk evictions. Container writable layers, container logs and emptyDir volumes all use the node's disk. A container with an ephemeral-storage limit that writes more than that limit is evicted:

yaml
resources:
  requests: { ephemeral-storage: 1Gi }
  limits:   { ephemeral-storage: 2Gi }

Fixes: set realistic memory requests, so the scheduler does not overpack nodes. Fix whatever is using too much. Send logs to standard output with rotation, instead of growing files (question 472). Clean up evicted pod records, which stay listed until deleted or garbage-collected.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet