Appearance
Resources, Scheduling and Scaling
389. What is the difference between resource requests and resource limits for CPU and memory?
The short answer
A request is the amount of CPU or memory a container is guaranteed, and the scheduler reserves it on a node when placing the pod. A limit is the most the container is allowed to use. Above its CPU limit, a container is slowed down, which is called throttling. Above its memory limit, it is killed. Requests decide where pods go and how CPU is shared under pressure. Limits decide what happens when a container tries to use more.
yaml
resources:
requests:
cpu: 250m # (1)
memory: 384Mi # (2)
limits:
memory: 512Mi # (3)- A quarter of a CPU core.
mmeans thousandths, so1000mis one full core. - 384 mebibytes.
MiandGiare powers of two: 1 MiB is 1,048,576 bytes. - The memory ceiling. Above 512 MiB, the kernel kills the container. There is no CPU limit here, which is a common and deliberate choice, explained below.
| Request | Limit | |
|---|---|---|
| Meaning | Guaranteed minimum | Hard maximum |
| Used by the scheduler | Yes | No |
| CPU above it | Allowed, if the node has spare | Throttled |
| Memory above it | Allowed, but evicted first under pressure | Killed (question 391) |
| Sets the QoS class | Yes, together with limits (question 393) | Yes |
Sensible defaults for a web service:
- Memory request at the app's normal working size, and memory limit equal to it or slightly above. Memory cannot be taken back gently, so a predictable ceiling is safer.
- CPU request at the app's typical use, so the scheduler places pods realistically.
- CPU limit: many teams leave it unset. Without it, a container can use idle CPU on the node, and is only held back, by its request's weight, when others need CPU too. With it, the container is throttled even when the node is idle (question 392). Some environments still require CPU limits, for example for cost control or strict fairness between tenants.
390. How does the scheduler use CPU and memory requests when placing a Pod?
The short answer
For each node, the scheduler adds up the requests of all the pods already placed there. It compares that total with the node's allocatable capacity. Allocatable capacity is the node's real capacity, minus what is set aside for the operating system and Kubernetes itself. A new pod fits only if its own requests fit in what is left. Actual current usage plays no part: a node can be nearly idle and still count as full if its pods requested a lot.
A worked example. A node has 4 CPUs and 16 GiB of memory.
| CPU | Memory | |
|---|---|---|
| Node capacity | 4000m | 16 GiB |
| Reserved for the system and Kubernetes | 180m | 1.5 GiB |
| Allocatable | 3820m | 14.5 GiB |
| Requests of pods already here | 3400m | 9 GiB |
| Free for scheduling | 420m | 5.5 GiB |
A new pod requesting 500m CPU and 1Gi memory does not fit: it needs 500m, only 420m is free. It does not matter that the existing pods are actually using only 1,200m right now.
bash
kubectl describe node aks-general-12345678-vmss000003 | grep -A 8 "Allocated resources"Allocated resources:
Resource Requests Limits
cpu 3400m (89%) 5200m (136%)
memory 9Gi (62%) 12Gi (82%)Limits can add up to more than 100%, which is called overcommitment: it works as long as not every pod uses its full limit at once.
Why requests, not usage? Usage changes every second. Scheduling on usage would pack pods onto a node during a quiet minute, and they would fight for resources at the next busy one. Requests give stable, predictable placement.
The consequences:
- Requests that are too high waste money: nodes look full while idle.
- Requests that are too low, or missing, overpack nodes: pods get evicted or slowed when everything gets busy. A pod with no requests counts as needing nothing, so the scheduler can put any number of them on one node.
391. What happens when a container exceeds its memory limit?
The short answer
The kernel kills it, because the limit is enforced by the container's cgroup. Kubernetes records the reason as OOMKilled, with exit code 137, and restarts the container according to the pod's restart policy, with the same limit. If it keeps happening, the pod goes into CrashLoopBackOff. This is different from an eviction, where the kubelet removes a pod because the whole node is short of memory.
What you see:
bash
kubectl get pod checkout-api-7d9f-abcde -n shop-prod
kubectl describe pod checkout-api-7d9f-abcde -n shop-prod | grep -A 6 "Last State"NAME READY STATUS RESTARTS AGE
checkout-api-7d9f-abcde 0/1 CrashLoopBackOff 5 (40s ago) 12m
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 27 Oct 2026 10:41:02 +0000
Finished: Mon, 27 Oct 2026 10:43:47 +0000What happened: the container's memory use hit its limit, the kernel's out-of-memory killer ended its process with SIGKILL, and the kubelet restarted it. The same mechanism as in plain Docker (question 97).
OOM kill versus eviction:
| OOM kill | Eviction | |
|---|---|---|
| Caused by | The container reaching its own limit | The node running short of memory |
| Done by | The kernel | The kubelet |
| What happens | The container restarts, same pod | The pod is removed; its controller makes a new one, maybe elsewhere |
| Shows as | OOMKilled, exit code 137 | Status Failed, reason Evicted (question 497) |
What to do is question 411: find out whether the app needs more memory, has a leak, or is sizing its own memory wrongly, before changing the limit.
392. What is CPU throttling, and how can it affect application latency?
The short answer
A CPU limit is enforced in short periods, 100 milliseconds by default: in each period, the container may use its limit's worth of CPU time, across all its threads. A multi-threaded app can use that allowance in the first part of the period, and is then frozen until the next period starts. That pause is throttling. Average CPU use can look low, yet requests that arrive during a frozen stretch wait, sometimes for tens of milliseconds, which shows up as slow responses at the high percentiles.
The arithmetic. A limit of 1 CPU means 100 ms of CPU time per 100 ms period. If four threads are busy at once, they consume 4 ms of budget every millisecond, so the budget is gone after 25 ms. The container is then frozen for the remaining 75 ms.
Why it surprises people. Over a whole minute, the container may average only 40% of its limit, so dashboards look healthy. But its work comes in bursts, such as a burst of requests, or garbage collection running on several threads, and each burst runs into the wall.
How to see it. The cgroup counts throttled periods (question 157). In Prometheus, the cAdvisor metrics container_cpu_cfs_throttled_periods_total divided by container_cpu_cfs_periods_total give the share of periods in which the container was throttled. More than a few percent on a latency-sensitive service is worth fixing.
How to reduce it:
- Remove the CPU limit, keeping a realistic CPU request. The request still protects other pods' fair share when the node is busy (question 389).
- Raise the limit, if your environment requires limits.
- Match the runtime's threads to the limit. For example, Go's
GOMAXPROCS, the number of threads running Go code at once, can be set to the container's CPU limit rather than the node's core count. Modern Java and newer Go versions read the container's limit automatically.
393. What are QoS classes such as Guaranteed, Burstable, and BestEffort, and how are they assigned?
The short answer
Kubernetes gives every pod a quality-of-service class based on its requests and limits. Guaranteed means every container has CPU and memory requests equal to its limits. BestEffort means no container has any requests or limits. Burstable is everything in between. The class decides who is removed first when a node runs out of memory: BestEffort pods first, then Burstable pods using more than they requested, and Guaranteed pods last.
| Class | Rule | Under node memory pressure |
|---|---|---|
| Guaranteed | Every container: CPU and memory requests equal limits | Evicted last |
| Burstable | Some requests or limits, but not Guaranteed | Evicted when using more than requested |
| BestEffort | No requests or limits at all | Evicted first |
yaml
# Guaranteed
resources:
requests: { cpu: "1", memory: 1Gi }
limits: { cpu: "1", memory: 1Gi }
---
# Burstable
resources:
requests: { cpu: 250m, memory: 384Mi }
limits: { memory: 512Mi }
---
# BestEffort: no resources section at allbash
kubectl get pod checkout-api-7d9f-abcde -o jsonpath='{.status.qosClass}'BurstableWhere the class is used:
- Eviction order when the node is short of memory or disk (question 497).
- The kernel's OOM priority. Kubernetes sets an OOM score adjustment for each container: BestEffort containers are the kernel's first choice to kill if the whole node runs out of memory, and Guaranteed containers the last.
Practical advice: important services should be Burstable with realistic requests, or Guaranteed for the most critical ones. Avoid BestEffort for anything that matters: it is the first to go, and it lets the scheduler overpack nodes.
394. How do taints, tolerations, node selectors, and node affinity influence Pod placement?
The short answer
Taints are set on nodes and repel pods: a tainted node only accepts pods that tolerate the taint. Tolerations are set on pods and allow them onto tainted nodes, but do not pull them there. Node selectors and node affinity are set on pods and attract them to nodes with certain labels, either as a hard rule or as a preference. To dedicate nodes to one kind of workload, combine both: a taint keeps others off, and affinity draws the right pods on.
Taints and tolerations: "keep out, unless".
bash
kubectl taint nodes gpu-node-1 gpu=true:NoScheduleyaml
tolerations:
- key: gpu
operator: Equal
value: "true"
effect: NoSchedule| Taint effect | Meaning |
|---|---|
NoSchedule | New pods without the toleration are not placed here |
PreferNoSchedule | Avoid placing them here if possible |
NoExecute | Also evicts running pods without the toleration, optionally after tolerationSeconds |
Kubernetes adds taints itself too. For example, a node that stops responding gets node.kubernetes.io/unreachable:NoExecute, which evicts its pods after a few minutes, so they are recreated elsewhere.
Node selectors and affinity: "go there".
yaml
nodeSelector:
node.kubernetes.io/instance-type: Standard_NC6s_v3 # (1)
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution: # (2)
nodeSelectorTerms:
- matchExpressions:
- { key: kubernetes.io/arch, operator: In, values: [amd64] }
preferredDuringSchedulingIgnoredDuringExecution: # (3)
- weight: 50
preference:
matchExpressions:
- { key: topology.kubernetes.io/zone, operator: In, values: [westeurope-1] }- A simple hard rule: only nodes with this label.
- Node affinity's hard form, with more operators, such as
In,NotIn,Exists,GtandLt. - A soft preference: zone 1 earns extra points in scoring, but other zones are allowed. "IgnoredDuringExecution" means a running pod is not moved if node labels change later.
Dedicated GPU nodes, the classic combination: taint the GPU nodes so ordinary pods stay off, and give GPU workloads both the toleration and an affinity for GPU nodes, so they go there and nowhere else.
395. What are pod affinity and anti-affinity, and when would you use them?
The short answer
Pod affinity places a pod near other pods with certain labels, and pod anti-affinity places it away from them, where "near" means within the same topology domain, such as the same node or the same zone. Use anti-affinity to keep copies of one service on different nodes or zones, so one failure does not take them all. Use affinity to put a pod close to something it talks to a lot, such as a cache, to cut network latency.
yaml
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels: { app.kubernetes.io/name: checkout-api }
topologyKey: kubernetes.io/hostname # (1)
podAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels: { app.kubernetes.io/name: checkout-cache }
topologyKey: topology.kubernetes.io/zone # (2)- Hard anti-affinity per node: never place two
checkout-apipods on the same node. If you ask for five replicas and have four nodes, the fifth staysPending. - Soft affinity per zone: prefer zones where a cache pod already runs.
The topologyKey is a node label that defines the domain: kubernetes.io/hostname means "the same node", and topology.kubernetes.io/zone means "the same availability zone".
Hard rules have sharp edges. Required anti-affinity can block scaling and rollouts when there are not enough domains, and it is relatively expensive for the scheduler to evaluate in large clusters. For simply spreading replicas evenly, topology spread constraints are usually the better tool (question 396).
396. What is a topology spread constraint, and how can it improve fault distribution?
The short answer
A topology spread constraint tells the scheduler to keep pods of a group evenly spread across domains, such as zones or nodes, within a tolerance called the maximum skew. With six replicas and three zones and a maximum skew of 1, you get two per zone, so losing one zone loses only a third of the capacity. You can make the rule strict, refusing to schedule rather than break it, or soft, preferring balance but allowing imbalance.
yaml
topologySpreadConstraints:
- maxSkew: 1 # (1)
topologyKey: topology.kubernetes.io/zone # (2)
whenUnsatisfiable: DoNotSchedule # (3)
labelSelector:
matchLabels: { app.kubernetes.io/name: checkout-api }
matchLabelKeys: [pod-template-hash] # (4)
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway # (5)
labelSelector:
matchLabels: { app.kubernetes.io/name: checkout-api }- The most allowed difference between the busiest and the emptiest domain. With 1, counts such as 2, 2, 2 or 3, 2, 2 are fine, and 3, 3, 0 is not.
- Balance across availability zones.
- Strict: if placing a pod would break the rule, leave it
Pending. - Count only pods from the same ReplicaSet. During a rollout, old and new pods are then balanced separately, which avoids lopsided placement once the old ones are gone.
- A second, softer rule: also spread across nodes when possible.
Why this beats anti-affinity for spreading: it allows more than one pod per domain while keeping them balanced, so it keeps working when replicas outnumber nodes or zones.
397. What is a Horizontal Pod Autoscaler (HPA), and which metrics can drive it?
The short answer
The Horizontal Pod Autoscaler changes the number of replicas of a Deployment or StatefulSet to keep a metric near a target. It can use three kinds of metric. Resource metrics, such as average CPU use as a percentage of the pods' requests. Custom metrics from the pods themselves, such as requests per second. Or external metrics, such as the length of a message queue. Its basic formula is: desired replicas equal current replicas times the current metric divided by the target, rounded up.
yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: checkout-api
namespace: shop-prod
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout-api }
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target: { type: Utilization, averageUtilization: 60 } # (1)
- type: Pods
pods:
metric: { name: http_requests_per_second }
target: { type: AverageValue, averageValue: "40" } # (2)
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # (3)
policies: [{ type: Percent, value: 50, periodSeconds: 60 }]- Keep average CPU at 60% of the pods' CPU requests.
- Also keep requests per second per pod at about 40. With several metrics, the HPA calculates a replica count for each and uses the largest.
- Scale down slowly. Act only on the lowest recommendation of the last five minutes, which is the default. Remove at most half the pods per minute. Then a short dip in traffic does not cause a big scale-down.
A worked example. Four replicas are running at 90% average CPU, against a target of 60%:
text
desired = ceil(4 × 90 / 60) = ceil(6) = 6 replicasThe HPA ignores small differences: by default, if the ratio is within 10% of the target, it does nothing, which prevents constant small changes.
Where the metrics come from:
| Kind | Source | Example |
|---|---|---|
| Resource | metrics-server | CPU, memory |
| Pods and Object, custom | An adapter, such as the Prometheus adapter | Requests per second |
| External | An adapter, or KEDA (question 496) | Queue length |
metrics-server is the small cluster add-on that collects CPU and memory use from each kubelet. It must be installed for CPU-based autoscaling, and for kubectl top, to work (question 635).
398. Why might HPA fail to scale a Deployment even when CPU usage is high?
The short answer
The most common reason is that the pods have no CPU request, so utilisation cannot be calculated and the HPA shows the current value as unknown. There are other reasons too. metrics-server may be missing or broken. The HPA may already be at its maximum replicas, or target the wrong workload. Its stabilisation rules may be holding it back. Or it did scale, but the new pods are stuck Pending, because the cluster has no room. kubectl describe hpa usually says which.
bash
kubectl get hpa checkout-api -n shop-prod
kubectl describe hpa checkout-api -n shop-prodNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
checkout-api Deployment/checkout-api <unknown>/60% 3 10 3
Conditions:
Type Status Reason Message
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container "api"The usual causes, and what shows:
| Cause | What you see |
|---|---|
| No CPU request on a container | <unknown> target; "missing request for cpu" |
| metrics-server missing or failing | <unknown>; kubectl top pods fails too |
Already at maxReplicas | Condition ScalingLimited with "the desired replica count is more than the maximum" |
Wrong scaleTargetRef | Events say the target cannot be found |
| Waiting on stabilisation | It scales, but later than expected |
| Scaled, but no room | Replicas went up; new pods Pending (question 408) |
| High CPU in only one pod | The average across pods is still below target |
| Pods not ready yet | Unready pods' metrics are treated cautiously during scale-up |
A detail about multi-container pods. CPU utilisation is calculated against the requests of all containers in the pod together. A sidecar with a large CPU request and little use can hide a busy app container. The ContainerResource metric type targets one container's CPU only.
And sometimes the HPA is right not to scale. Sometimes the work is limited by something other than CPU, such as a database or a lock. Then more pods do not help, and the right metric to scale on is a different one.
399. How do Pod autoscaling and cluster/node autoscaling differ?
The short answer
Pod autoscaling changes how many pods run, with the HPA, or how big each pod's requests are, with the VPA, inside the room the cluster already has. Node autoscaling changes how much room there is: it adds nodes when pods cannot be scheduled for lack of resources, and removes nodes that are underused. They work as a chain: the HPA adds pods, some cannot fit and stay Pending, the node autoscaler sees them and adds nodes, and the pods start there.
| Pod autoscaling | Node autoscaling | |
|---|---|---|
| Changes | Number of pods, or their size | Number of nodes |
| Tools | HPA, VPA, KEDA | Cluster Autoscaler, Karpenter, cloud node auto-provisioning |
| Triggered by | Metrics such as CPU or queue length | Pods Pending for lack of resources; underused nodes |
| Speed | Seconds | Minutes, while a machine starts |
Two node autoscalers you will hear about:
- Cluster Autoscaler: works with node groups, such as AKS node pools or EKS node groups, each with a fixed machine type. It grows the group that can fit the pending pods, and shrinks groups whose nodes are underused and whose pods can move elsewhere.
- Karpenter: started at AWS, now also used for Azure through its node auto-provisioning feature. It looks at the pending pods' actual needs, such as CPU, memory, architecture and zone, and creates a suitably sized machine directly, often choosing cheaper or better-fitting types. It also consolidates, replacing several half-empty nodes with fewer ones.
Scale-down needs care. Removing a node means evicting its pods. The node autoscaler respects PodDisruptionBudgets (question 400), and pods with local storage or special annotations can prevent a node from being removed.
The timing gap. A traffic spike can outrun node provisioning. Common answers are a little spare capacity, often provided by low-priority placeholder pods that are evicted to make room, and scaling on a leading signal, such as queue length, rather than CPU.
400. What is a PodDisruptionBudget, and what does it protect during voluntary disruptions?
The short answer
A PodDisruptionBudget limits how many pods of a group can be taken down at the same time by voluntary actions, such as draining a node for maintenance, a node autoscaler removing a node, or a cluster upgrade. You set either the minimum number that must stay available or the maximum that may be unavailable. Evictions that would break the budget are refused and retried later. It does not protect against involuntary failures, such as a node crashing.
yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-api
namespace: shop-prod
spec:
maxUnavailable: 1 # (1)
selector:
matchLabels: { app.kubernetes.io/name: checkout-api }
unhealthyPodEvictionPolicy: AlwaysAllow # (2)- At most one
checkout-apipod may be evicted at a time.minAvailable: 2would express a similar rule from the other side. - Pods that are already unhealthy may always be evicted, so a broken pod cannot block maintenance.
Voluntary versus involuntary:
| Voluntary, PDB applies | Involuntary, PDB does not apply |
|---|---|
kubectl drain for node maintenance | Node hardware failure |
| Node autoscaler scale-down | Kernel panic, VM deleted |
| Managed cluster upgrades | Out-of-memory kill |
| Evictions through the Eviction API | Network partition |
PDBs work through the Eviction API: tools that ask to evict a pod politely are told "not now" if the budget would be broken. Deleting a pod directly with kubectl delete pod bypasses the budget.
Budgets that block everything:
minAvailableequal to the replica count, ormaxUnavailable: 0, means no voluntary eviction can ever happen, so node drains and upgrades hang forever (question 426).- A single-replica app with
minAvailable: 1has the same effect. Either run two replicas, or accept the brief disruption.
PDBs and rollouts are separate. A Deployment's rolling update is controlled by maxUnavailable in its own strategy, not by the PDB.
493. What are ResourceQuotas and LimitRanges?
The short answer
A ResourceQuota caps what a whole namespace may use in total: for example, at most 20 CPUs of requests, 64 GiB of memory limits, 100 pods and 2 load balancers. A LimitRange sets rules for each individual container or pod in a namespace: default requests and limits for containers that do not set their own, and minimum and maximum values. Together they stop one team or one mistake from consuming a shared cluster, and make sure every pod gets sensible resource settings.
yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: shop-dev-quota
namespace: shop-dev
spec:
hard:
requests.cpu: "20"
requests.memory: 48Gi
limits.memory: 64Gi
pods: "100"
persistentvolumeclaims: "20"
services.loadbalancers: "2" # (1)
---
apiVersion: v1
kind: LimitRange
metadata:
name: shop-dev-defaults
namespace: shop-dev
spec:
limits:
- type: Container
defaultRequest: { cpu: 100m, memory: 128Mi } # (2)
default: { memory: 256Mi } # (3)
max: { memory: 2Gi } # (4)- Load balancers cost money, so a quota on them is common.
- Containers with no requests get these, added at admission.
- Containers with no limit get a 256 MiB memory limit.
- No single container may ask for more than 2 GiB.
How they work together. When a namespace has a quota on CPU or memory, every new pod must declare those resources, or it is rejected. A LimitRange's defaults fill them in automatically, so teams are not blocked by forgetting.
When the quota is full, new pods are rejected at admission, not left pending:
Error creating: pods "checkout-api-5c8b-x2k9" is forbidden: exceeded quota: shop-dev-quota, requested: requests.cpu=500m, used: requests.cpu=19800m, limited: requests.cpu=20That message often appears only in the ReplicaSet's events, not on any pod, because the pod was never created (question 428).
494. What are PriorityClasses and preemption?
The short answer
A PriorityClass gives pods a priority number, and the higher the number, the more important the pod. When a high-priority pod cannot be scheduled because the cluster is full, the scheduler may preempt lower-priority pods: evict them to make room. Priority also affects which pods are evicted first when a node runs short of resources. Use it to protect critical services over batch jobs, carefully, because misused priorities let one workload push others out.
yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: shop-critical
value: 100000 # (1)
description: Customer-facing checkout path
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: batch-low
value: 1000
preemptionPolicy: Never # (2)
---
# in a pod spec:
priorityClassName: shop-critical- A bigger number means a more important pod. Kubernetes reserves very large values, around two billion, for its own
system-cluster-criticalandsystem-node-criticalclasses, used by components such as DNS. - Pods of this class never evict others to get scheduled. They wait instead.
How preemption works:
- A
shop-criticalpod isPending: no node has room. - The scheduler looks for a node where evicting some lower-priority pods would make enough room.
- It evicts them, giving them their normal graceful shutdown, and records which node it chose for the waiting pod.
- The critical pod is scheduled there once the room is free.
Uses:
- Checkout and payment services above internal tools and batch jobs.
- Placeholder pods with negative or very low priority, which hold spare capacity and are instantly preempted when real pods need room (question 399).
Risks: if every team marks its pods as highest priority, priority means nothing. And preemption ignores PodDisruptionBudgets when it has no other choice, so treat it as a last resort, not normal scheduling.
495. What is the Vertical Pod Autoscaler, and can it run together with the Horizontal Pod Autoscaler?
The short answer
The Vertical Pod Autoscaler watches how much CPU and memory pods really use over time, and recommends, or sets, better requests for them. In its recommendation-only mode, it simply tells you good values. In its active modes, it applies them, traditionally by recreating pods. It should not manage the same CPU or memory metric that an HPA is scaling on, because the two would fight. VPA on requests plus an HPA on a different metric, such as requests per second, works well.
yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: checkout-api
namespace: shop-prod
spec:
targetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout-api }
updatePolicy:
updateMode: "Off" # (1)
resourcePolicy:
containerPolicies:
- containerName: api
minAllowed: { cpu: 100m, memory: 256Mi }
maxAllowed: { cpu: "2", memory: 2Gi }- Recommendation only.
kubectl describe vpa checkout-apithen shows a recommended request for each container, based on its real usage history.
| Update mode | Behaviour |
|---|---|
Off | Recommends only |
Initial | Sets requests when pods are created, never changes them after |
Recreate | Evicts pods whose requests are far from the recommendation, so they come back with new values |
InPlaceOrRecreate | Uses in-place resizing where the cluster supports it, otherwise recreates |
In-place resizing. Newer Kubernetes versions can change a running pod's CPU and memory without restarting it. VPA can use that, which removes most of the disruption of its active modes.
Why VPA and HPA conflict on the same metric. Suppose the HPA keeps CPU at 60% of requests, and VPA raises the requests because usage is high. Utilisation, being usage divided by requests, drops, so the HPA removes pods, so usage per pod rises, so VPA raises requests again. The two keep undoing each other.
A safe combination: VPA in recommendation mode to set sensible requests during reviews, plus an HPA scaling on CPU or on a custom metric.
496. What is KEDA, and when would you scale on something like queue length?
The short answer
KEDA, Kubernetes Event-driven Autoscaling, scales workloads based on outside event sources, such as the number of messages waiting in a queue, the lag of a Kafka consumer group, a Prometheus query or a schedule. It can scale a workload all the way to zero when there is nothing to do, and back up when work arrives. Scale on queue length when the queue, not CPU, is the true measure of how much work is waiting, which is the case for most background workers.
The shop's notification worker reads "order placed" events from an Azure Service Bus queue. Its CPU is low even when thousands of emails are waiting, because each email mostly waits on the email provider. CPU-based scaling would never add workers. The queue length tells the real story.
yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: notification-worker
namespace: shop-prod
spec:
scaleTargetRef: { name: notification-worker } # (1)
minReplicaCount: 0 # (2)
maxReplicaCount: 30
triggers:
- type: azure-servicebus
metadata:
queueName: order-placed
namespace: shop-bus
messageCount: "50" # (3)
authenticationRef: { name: servicebus-workload-identity } # (4)- The Deployment to scale.
- Scale to zero when the queue is empty, which saves money for workloads that are idle much of the time.
- Aim for about 50 waiting messages per replica. With 1,000 messages waiting, KEDA asks for 20 replicas.
- KEDA reads the queue's length using workload identity, without stored passwords.
How it works. KEDA runs a small operator and a metrics adapter. For each ScaledObject, it creates and manages a normal HPA fed with the external metric, and it handles the special step between zero and one replica itself, because a normal HPA cannot scale to zero.
Good fits: queue workers, stream consumers, scheduled bursts, and anything whose demand is measured outside the pods. For run-to-completion work, KEDA's ScaledJob creates Jobs instead of scaling a Deployment.
497. What are node-pressure evictions, and why do pods show up as Evicted?
The short answer
When a node runs low on memory, disk space or process IDs, the kubelet evicts pods to protect the node itself, choosing first those using the most above their requests, and lower-priority, lower-QoS pods before others. An evicted pod shows status Failed with reason Evicted and a message saying which resource ran low, and its controller creates a replacement, perhaps on another node. Containers writing too much to their local disk can also be evicted for exceeding their ephemeral-storage limit.
What the kubelet watches, with typical default thresholds:
| Signal | Typical hard threshold |
|---|---|
memory.available | Below 100 MiB |
nodefs.available, the node's main disk | Below 10% |
imagefs.available, the disk holding images | Below 15% |
pid.available | Close to the node's process limit |
Clusters can also set soft thresholds with grace periods, so short spikes do not trigger evictions. Managed services set their own values.
Who goes first:
- Pods using more than their request of the scarce resource.
- Among those, lower priority first (question 494).
- Then by how far above their request they are.
In practice, BestEffort pods go first and Guaranteed pods last (question 393).
What it looks like:
bash
kubectl get pods -n shop-prod --field-selector=status.phase=Failed
kubectl describe pod report-builder-6d4f-pq2xz -n shop-prod | grep -A 2 ReasonReason: Evicted
Message: The node was low on resource: memory. Threshold quantity: 100Mi, available: 87Mi. Container builder was using 2804Mi, request is 512Mi, has larger consumption of memory.The message names the guilty container and how far it was above its request.
Disk evictions. Container writable layers, container logs and emptyDir volumes all use the node's disk. A container with an ephemeral-storage limit that writes more than that limit is evicted:
yaml
resources:
requests: { ephemeral-storage: 1Gi }
limits: { ephemeral-storage: 2Gi }Fixes: set realistic memory requests, so the scheduler does not overpack nodes. Fix whatever is using too much. Send logs to standard output with rotation, instead of growing files (question 472). Clean up evicted pod records, which stay listed until deleted or garbage-collected.