Appearance
Security and Day-2 Operations
415. What is RBAC, and how do Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings differ?
The short answer
RBAC, role-based access control, decides who may do what in the Kubernetes API. A Role is a list of allowed actions on resources inside one namespace. A ClusterRole is the same, but can cover cluster-wide resources, such as nodes, or be reused in many namespaces. A RoleBinding gives a Role or a ClusterRole to users, groups or service accounts within one namespace. A ClusterRoleBinding gives a ClusterRole across the whole cluster. Permissions only add up: there is no "deny".
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deployment-reader
namespace: shop-prod
rules:
- apiGroups: ["apps"] # (1)
resources: ["deployments"]
verbs: ["get", "list", "watch"] # (2)
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: checkout-team-reads-deployments
namespace: shop-prod
subjects:
- kind: Group
name: checkout-team # (3)
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: deployment-reader
apiGroup: rbac.authorization.k8s.io- Deployments belong to the
appsAPI group. Core resources such as pods use"". - Verbs are API actions:
get,list,watch,create,update,patch,delete. - The subject: a group of people, coming from the cluster's login provider. Subjects can also be single users or service accounts.
Built-in ClusterRoles worth knowing: view (read most things, but not Secrets), edit (change most things in a namespace), admin (everything in a namespace, including its roles), and cluster-admin (everything everywhere).
Checking permissions:
bash
kubectl auth can-i delete pods -n shop-prod
kubectl auth can-i --list -n shop-prod --as=system:serviceaccount:shop-prod:checkout-api416. How do you grant a service account only the permissions an application needs?
The short answer
Give each application its own service account, never the namespace's default one. If the app never calls the Kubernetes API, turn off mounting its token. If it does, create a Role listing exactly the resources, names and verbs it uses, bind it in the app's own namespace, and test it with kubectl auth can-i --as. Avoid cluster-wide bindings and wildcards. For cloud access, map the service account to a cloud identity with equally narrow permissions.
Most apps need no API access at all:
yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: checkout-api
namespace: shop-prod
automountServiceAccountToken: false # (1)- No API token is mounted into the pods, so a compromised app has no credentials for the cluster's API.
An app that does need access, for example a worker that reads one ConfigMap and creates Jobs:
yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: report-scheduler
namespace: shop-prod
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["report-schedule"] # (1)
verbs: ["get", "watch"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "get", "list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: report-scheduler
namespace: shop-prod
subjects:
- kind: ServiceAccount
name: report-scheduler
namespace: shop-prod # (2)
roleRef: { kind: Role, name: report-scheduler, apiGroup: rbac.authorization.k8s.io }- Only that one ConfigMap, by name.
- The service account's own namespace must be stated in the subject. A wrong namespace here is a common reason a binding "does nothing".
Test it:
bash
kubectl auth can-i create jobs -n shop-prod --as=system:serviceaccount:shop-prod:report-scheduler # yes
kubectl auth can-i get secrets -n shop-prod --as=system:serviceaccount:shop-prod:report-scheduler # noThings to avoid: wildcards such as verbs: ["*"], permissions on Secrets unless truly needed, create on pods (which lets the app mount any Secret into a new pod), and any ClusterRoleBinding for an ordinary app.
Cloud access is separate. If the app needs, say, a storage bucket, bind its service account to a cloud identity through workload identity, with permission for that bucket only (question 615).
417. What is the difference between a Kubernetes service account and a human user's identity?
The short answer
A service account is an identity for software running in the cluster. It is a real Kubernetes object in a namespace, and Kubernetes itself issues its tokens, which are short-lived and mounted into pods automatically. Human users are not stored in Kubernetes at all: people are authenticated by an outside identity provider, such as Microsoft Entra ID or Google, through single sign-on, and Kubernetes only sees their name and groups. RBAC can grant permissions to both, but they are managed in completely different places.
| Service account | Human user | |
|---|---|---|
| Stored in Kubernetes | Yes, as an object | No |
| Created with | kubectl create serviceaccount | The company's identity provider |
| Credentials | Short-lived tokens issued by Kubernetes | Single sign-on tokens, sometimes client certificates |
| Used by | Pods, controllers, pipelines | People, through kubectl |
| Name in logs | system:serviceaccount:shop-prod:checkout-api | asha@shop.example.com |
| Grouped by | Namespace | Groups from the identity provider |
Service account tokens today. Pods receive a projected token: it is valid for a limited time, typically about an hour, is meant for one audience, is tied to that specific pod, and is refreshed by the kubelet automatically. The old style, long-lived tokens stored in Secrets, is discouraged.
Human access done well:
- Single sign-on with multi-factor authentication, through the cluster's OIDC integration or the cloud's own, such as Entra ID for AKS.
- Group-based permissions: bind roles to groups such as
checkout-team, never to individuals. When someone leaves, removing them from the company directory removes their cluster access. - No shared accounts, and no people using service account tokens, so the audit log always shows who did what.
418. Why should application containers avoid running as root, and which Pod security settings help enforce that?
The short answer
Root in a container is root on the node's kernel, unless user namespaces are used, so any escape bug or risky mount gives an attacker control of the node. The pod's securityContext lets you refuse to run as root, set a specific non-root user, stop privilege escalation, drop capabilities, use a read-only filesystem and apply a system call filter. Pod Security Admission's restricted level, or a policy engine, then makes those settings mandatory for a whole namespace.
Question 139 explains why root is dangerous. In Kubernetes, the settings are:
yaml
spec:
securityContext: # applies to every container in the pod
runAsNonRoot: true # (1)
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001 # (2)
seccompProfile: { type: RuntimeDefault } # (3)
containers:
- name: api
securityContext: # this container only
allowPrivilegeEscalation: false # (4)
readOnlyRootFilesystem: true
capabilities: { drop: [ALL] } # (5)- The kubelet refuses to start any container that would run as user 0. With a numeric user in the image, or
runAsUserhere, it can check this before starting. - Volumes that support ownership are made accessible to group 10001, so the non-root user can write to them.
- The container runtime's default system call filter (question 419).
- A process cannot gain more privileges than its parent, for example through programs marked
setuid. - Removes every Linux capability (question 101).
Enforcing it for a namespace:
- Pod Security Admission at the
restrictedlevel requires most of these settings (question 499). - Policy engines, such as Kyverno, can require exactly your organisation's rules (question 500).
Going further: user namespaces. With hostUsers: false in the pod spec, the pod runs in its own user namespace, so even root inside it is an unprivileged user on the node (question 142).
419. What are Linux capabilities, seccomp profiles, and read-only root filesystems in Kubernetes security?
The short answer
They are three container hardening settings, each set in the security context. Capabilities split root's power into pieces: dropping all of them and adding back only what is needed, such as permission to use low ports, limits what even a root process can do. A seccomp profile filters the system calls a container may make: RuntimeDefault blocks dangerous, rarely needed ones. A read-only root filesystem stops anything, including an attacker, from changing files in the container, with small writable volumes where the app truly needs them.
| Setting | Field | Recommended value |
|---|---|---|
| Capabilities | securityContext.capabilities | drop: [ALL], then add only what is needed |
| Seccomp | securityContext.seccompProfile | type: RuntimeDefault |
| Read-only root filesystem | securityContext.readOnlyRootFilesystem | true, plus emptyDir for writable paths |
| AppArmor | securityContext.appArmorProfile | type: RuntimeDefault, where available |
Capabilities (question 101):
yaml
securityContext:
capabilities:
drop: [ALL]
add: [NET_BIND_SERVICE] # only if the app must listen on a port below 1024Most apps need none: listen on 8080 and let the Service map port 80 to it.
Seccomp profiles:
| Type | Meaning |
|---|---|
RuntimeDefault | The container runtime's standard filter: blocks dozens of risky system calls |
Localhost | A custom profile file on the node, for very strict lock-down |
Unconfined | No filter at all; avoid |
Without any setting, many clusters run containers unconfined, so stating RuntimeDefault explicitly matters.
A read-only root filesystem, with writable spots:
yaml
containers:
- name: api
securityContext: { readOnlyRootFilesystem: true }
volumeMounts:
- { name: tmp, mountPath: /tmp }
volumes:
- name: tmp
emptyDir: { sizeLimit: 64Mi }The app can write to /tmp and nowhere else. If it fails to start with "read-only file system", its error message shows which path it needs (question 96).
420. How do image-pull secrets differ from application secrets?
The short answer
An image-pull secret holds registry credentials, and only the kubelet uses it, to download the container's image before the container starts. It is never visible to the app. An application secret holds what the app itself needs while running, such as a database password or an API key, and is delivered into the container as a file or a variable. They have different users, different permissions and different lifetimes, and should never be mixed.
| Image-pull secret | Application secret | |
|---|---|---|
| Type | kubernetes.io/dockerconfigjson | Usually Opaque |
| Used by | The kubelet, before the container starts | The app, while running |
| Referenced by | imagePullSecrets on the pod or service account | env, envFrom or a volume |
| Visible inside the container | No | Yes |
| Should allow | Reading images from one registry | Exactly what the app needs |
| Better alternative | The node's cloud identity (question 354) | An outside secret manager (question 492) |
Why the separation matters. Suppose a registry credential were given to the app as a variable. Then any compromise of the app would also leak access to your registry. If the credential allowed it, the attacker could even push images. Keeping pull credentials read-only and only in the kubelet's hands keeps that door shut (question 132).
421. How would you investigate a Pod that cannot access an API because of authorization errors?
The short answer
First find out which API refused it, and with which code. A 401 means the credentials were missing or invalid. A 403 means the identity was recognised but not allowed. For the Kubernetes API, the error names the service account and the action it was denied. Check which service account the pod uses, and whether its token is mounted. Then check whether a binding in the right namespace gives a role with the right resource, API group and verb. For cloud APIs, check the workload identity set-up and the cloud role assignment.
The Kubernetes API's 403 message tells you almost everything:
Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:shop-prod:report-scheduler" cannot create resource "jobs" in API group "batch" in the namespace "shop-prod"It names the identity, the verb, the resource, the API group and the namespace.
Check, in order:
- Which service account does the pod use?
kubectl get pod NAME -o jsonpath='{.spec.serviceAccountName}'. It may be thedefaultone by accident. - Is a token mounted? If
automountServiceAccountTokenis false, the app has no credentials, and the API answers401. - Does a binding exist?
kubectl get rolebinding -n shop-prod -o widelists bindings and their subjects. Check that the subject's namespace is the service account's own namespace. - Does the role allow this? Wrong API groups are the classic mistake: Deployments and Jobs are not in the core group
"", they are inappsandbatch. Cluster-scoped resources, such as nodes, need a ClusterRole and a ClusterRoleBinding. - Test the exact action:
kubectl auth can-i create jobs.batch -n shop-prod --as=system:serviceaccount:shop-prod:report-scheduler.
For cloud APIs, such as storage or Key Vault, the denial comes from the cloud, not Kubernetes:
- Is workload identity set up, with the service account annotated with the right cloud identity (question 615)?
- Does the federated credential's subject match
system:serviceaccount:shop-prod:report-schedulerexactly? - Does the cloud identity have a role on the specific resource?
- Has enough time passed? Cloud role assignments can take a few minutes to apply.
422. How do you safely roll out a new configuration without accidentally restarting every service at once?
The short answer
Treat configuration like code: version it, and roll it out per service, gradually, with health checks, rather than changing something shared that every service reads at once. Give each version of a ConfigMap a new name, so a change becomes a normal rolling update of one Deployment. Try it first in a non-production environment or on a canary. Use feature flags for behaviour changes. And spread changes across services and clusters over time, so one bad value cannot hit everything together.
The dangerous pattern: one big shared ConfigMap, read by twenty services, with a reloader that restarts everything that uses it on any change. One edit restarts twenty services at the same moment, and if the new value is wrong, all twenty break together.
Safer patterns:
- Versioned names. Kustomize's
configMapGenerator, or a hash in the pod template, turns a configuration change into a normal rolling update of that one Deployment, withmaxUnavailable, readiness checks and rollback (question 380). - One owner per configuration. Each service has its own ConfigMap. Shared values go through a shared library or a configuration service with its own gradual rollout.
- Staged rollout. Development, then staging, then a canary in production, then everywhere, the same as for code (question 133).
- Feature flags for behaviour. "Enable the new pricing engine" is a flag rolled out to a percentage of users, not a ConfigMap edit (question 285).
- Spread across clusters. With GitOps across several clusters, roll configuration out to one cluster first, then the rest.
- Limits on blast radius. PodDisruptionBudgets and
maxUnavailablecap how many pods change at once, even if something does trigger many rollouts.
423. How would you determine whether a production incident is caused by the application, container image, Pod configuration, node, network, storage, or cluster control plane?
The short answer
Narrow it down by scope and by timing. Ask what is affected: one pod, one node, one service, one zone or everything. Ask what changed just before: a deployment, a configuration change, a node upgrade or nothing. Then let the pattern point at the layer. Failures that follow one node point at the node. Failures that follow a new image point at the image or the code. Name lookups and timeouts point at the network. Attach and slowness errors point at storage. A slow or failing kubectl points at the control plane.
Questions, in order:
- What changed? Deployments, configuration commits, node pool upgrades, certificate renewals, cloud changes. Most incidents follow a change. If one lines up, roll it back first and investigate afterwards (question 456).
- What is the scope?
| Affected | Layer to suspect |
|---|---|
| One pod | That pod: its node, its data, a bad request |
| All pods on one node | The node: pressure, kernel, network on that node |
| One service on every node | The app, its image or its configuration |
| Several services in one zone | The zone: network, storage, cloud problems |
| Everything | Shared parts: DNS, the gateway, the control plane, the cloud |
- Which signals?
| Symptom | Points to |
|---|---|
| Errors in the app's logs, stack traces | Application |
| New image digest started at the same time | Image or code |
| Container fails to start, permission or file errors | Pod configuration |
NotReady, evictions, pressure conditions | Node (question 498) |
| "no such host", connection timeouts | Network or DNS (question 375) |
| Volume attach errors, high disk latency | Storage |
kubectl slow or timing out, controllers lagging | Control plane |
- Confirm with a direct test at that layer before acting: a DNS lookup from an affected pod, a request to the app from inside its pod, the node's conditions, the volume's events.
424. What is the difference between application logs, Kubernetes events, container runtime logs, and node logs?
The short answer
Application logs are what your app writes to standard output and error, read with kubectl logs. Kubernetes events are short records the cluster keeps about what it did to objects, such as scheduling, pulling images, killing containers, and they expire after about an hour. Container runtime logs are containerd's own records of pulling images and creating containers, kept on each node. Node logs are the kubelet's and the kernel's logs on each machine, which explain things like out-of-memory kills, disk problems and kubelet failures.
| Kind | Written by | Tells you | Where to find it |
|---|---|---|---|
| Application logs | Your app | What it did and what went wrong inside it | kubectl logs; files under /var/log/pods on the node |
| Kubernetes events | Scheduler, controllers, kubelet | What the cluster did to an object | kubectl describe, kubectl get events |
| Container runtime logs | containerd | Pulls, sandbox and container start errors | journalctl -u containerd on the node |
| Node logs | kubelet, kernel, system services | Probe kills, evictions, OOM kills, disk and network trouble | journalctl -u kubelet, dmesg on the node |
An example of why you need several. A pod restarts:
- The app log ends abruptly with no error.
- The events show "Container api was killed".
- The kubelet log shows it was killed for failing its liveness probe.
- The kernel log would instead show "Memory cgroup out of memory: Killed process" if it had been an OOM kill.
Keep them centrally. Events expire quickly and node logs disappear with the node. A log collector, such as Fluent Bit running as a DaemonSet, sends application logs to a central store, and tools such as the Kubernetes event exporter copy events there too. Managed clusters can also send their control-plane and node logs to the cloud's log service (question 633).
425. What basic metrics would you monitor for Pod health, resource saturation, node capacity, and rollout health?
The short answer
For pods: restarts, readiness, OOM kills, and CPU and memory use compared with requests and limits, including CPU throttling. For nodes: requested versus allocatable resources, pressure conditions, and NotReady nodes. For capacity: pods stuck Pending and autoscaler activity. For rollouts: available versus desired replicas, rollout duration, and the error rate and latency of the new version against the old. Most of these come from kube-state-metrics, the kubelet's own metrics, node-exporter and the app.
| Area | Metric | Why |
|---|---|---|
| Pod health | Container restarts per hour | Crashes, kills, probe failures |
| Pods not ready | Removed from traffic | |
| OOM-killed containers | Memory limits hit | |
| Saturation | CPU use against requests and limits | Sizing, and approaching limits |
| CPU throttled periods | Latency caused by CPU limits (question 392) | |
| Memory working set against limits | Risk of OOM kills | |
| Node capacity | Sum of requests against allocatable, per node | How full the scheduler thinks nodes are |
| Node pressure conditions, NotReady nodes | Unhealthy machines | |
| Pending pods | Not enough room, or scheduling rules that cannot be met | |
| Rollouts | Available replicas against desired | Progress and stuck rollouts |
| Error rate and p99 latency by version | Whether the new version is worse (question 284) |
Where they come from:
- kube-state-metrics: turns the state of Kubernetes objects, such as desired replicas, pod phases and restart counts, into metrics.
- The kubelet and its built-in cAdvisor: CPU, memory and throttling per container.
- node-exporter: CPU, memory, disk and network per machine.
- The app itself: requests, errors and latency.
Question 635 explains how these fit together with metrics-server and Prometheus.
Alert on symptoms, watch causes. Page someone for "checkout errors above target" or "pods Pending for 10 minutes". Keep "node CPU at 80%" on dashboards, not as a 3 a.m. page (question 628).
426. How do you safely drain a node for maintenance, and what could prevent the drain from completing?
The short answer
Cordon the node so nothing new is scheduled there, then drain it: Kubernetes evicts its pods one by one, respecting their disruption budgets and shutdown grace periods, and their controllers recreate them on other nodes. The drain can get stuck in several ways. A disruption budget may allow no evictions. Some pods may not be managed by any controller, or may use local emptyDir data. Pods may be stuck terminating. Or the rest of the cluster may have no room for the evicted pods.
bash
kubectl cordon aks-general-12345678-vmss000003 # (1)
kubectl drain aks-general-12345678-vmss000003 \
--ignore-daemonsets --delete-emptydir-data --timeout=15m # (2)
# ...maintenance...
kubectl uncordon aks-general-12345678-vmss000003 # (3)- Marks the node unschedulable. Running pods stay for now.
- Evicts every pod through the Eviction API.
--ignore-daemonsetsskips DaemonSet pods, which would only be recreated on the same node.--delete-emptydir-dataaccepts thatemptyDircontents will be lost. The timeout stops it waiting forever. - Makes the node schedulable again after maintenance.
What blocks a drain:
| Blocker | Message or sign | Fix |
|---|---|---|
| A disruption budget allows no eviction | "Cannot evict pod as it would violate the pod's disruption budget" | Wait for other pods to be ready, or fix an over-strict budget (question 400) |
| Pod with no controller | "cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet" | --force deletes it for good; better, put it under a controller |
Pod with emptyDir data | Refuses without the flag | --delete-emptydir-data, if losing the data is acceptable |
Pod stuck Terminating | Waits for a long grace period or a finalizer | Find out why it does not stop (question 487) |
| No room elsewhere | Evicted pods go Pending | Add capacity first, or let the autoscaler |
| Single-replica app with a strict budget | Eviction refused forever | Two replicas, or accept the disruption |
Managed node pool upgrades use the same mechanism. They usually add a surge node first, then drain old nodes one at a time, so the cluster never loses capacity (question 501).
427. How do you roll back a Deployment after discovering a bad image or configuration change?
The short answer
For a bad image or pod template change, kubectl rollout undo returns the Deployment to its previous revision through a normal rolling update. If the cluster is managed with GitOps, revert the change in Git instead, or the GitOps tool will put the bad version back. If the bad change was in a ConfigMap or Secret, rolling back the Deployment alone does not help, unless configuration is versioned with the pods, so restore the configuration too. Then confirm the running version and the health signals.
A bad image or pod template:
bash
kubectl rollout undo deploy/checkout-api -n shop-prod
kubectl rollout status deploy/checkout-api -n shop-prod --timeout=5mWith GitOps: revert the commit that changed the digest, merge it, and let Argo CD or Flux sync. Git stays the record of what runs (question 512).
With Helm: helm rollback checkout-api 41 -n shop-prod restores revision 41 of the whole release, including its ConfigMaps and other objects, not just the pod template (question 505).
A bad configuration change:
- If the ConfigMap is versioned, with a hash in its name, the Deployment's previous revision points at the previous ConfigMap, so
rollout undorestores both together. That is one of the main reasons to version configuration (question 380). - If one ConfigMap was edited in place, the old values are gone from the cluster. Restore them from Git, then roll the pods so they pick them up.
After any rollback, verify: the pods run the expected digest, the readiness of all replicas, and error rate and latency back to normal (question 230).
And remember the database. If the bad release changed the schema, rolling back the application may not be enough, or even safe (question 286).
428. What would you check if a rollout remains stuck and old Pods are not being replaced?
The short answer
Look at the new ReplicaSet's pods, because old pods are only removed as new ones become available. They may be Pending for lack of resources, failing to pull their image, crash-looping, or running but never ready. If no new pods exist at all, look at the ReplicaSet's events: a quota or an admission policy may be refusing to create them. Also check whether the rollout is paused, and the Deployment's Progressing condition, which reports when the progress deadline has passed.
bash
kubectl rollout status deploy/checkout-api -n shop-prod # (1)
kubectl get rs -n shop-prod -l app.kubernetes.io/name=checkout-api # (2)
kubectl describe deploy checkout-api -n shop-prod | sed -n '/Conditions/,/Events/p' # (3)
kubectl describe rs checkout-api-5c8b -n shop-prod | tail -10 # (4)- Shows how far it got: "Waiting for deployment rollout to finish: 1 out of 3 new replicas have been updated".
- The old and new ReplicaSets with their desired, current and ready counts.
- The
Progressingcondition:ProgressDeadlineExceededmeans no progress forprogressDeadlineSeconds. - The new ReplicaSet's events, which show refusals to create pods.
Match what you find:
| Finding | Cause |
|---|---|
New pods Pending | No room, or scheduling rules (question 408) |
New pods ImagePullBackOff | Wrong image or credentials (question 406) |
New pods CrashLoopBackOff | The new version crashes (question 407) |
New pods running, 0/1 ready | Readiness failing (question 409) |
| No new pods; ReplicaSet events say "exceeded quota" | Namespace quota full (question 493) |
| No new pods; events say "admission webhook ... denied" | A policy rejects the new pod spec (question 500) |
Deployment shows Paused | Someone paused it: kubectl rollout resume |
| New pods ready, but slowly counted | A long minReadySeconds |
Why old pods stay. With maxUnavailable: 0, Kubernetes never removes an old pod until a new one is available, so a broken new version is held back while the old version keeps serving. That is the design working correctly. The rollout does not roll itself back, though: fix the cause, or run kubectl rollout undo.
429. How would you prevent disruption while restarting or upgrading a workload that has only two replicas?
The short answer
Make sure one replica is always serving. Roll out with maxUnavailable: 0 and maxSurge: 1, so a new pod must be ready before an old one stops. Add a disruption budget allowing at most one unavailable pod, so drains and upgrades take them one at a time. Keep the two replicas on different nodes and zones. Use honest readiness probes and a graceful shutdown with a preStop pause. And consider running three, because with two, losing one halves your capacity.
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog-api
spec:
replicas: 2
strategy:
rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } # (1)
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule # (2)
labelSelector: { matchLabels: { app.kubernetes.io/name: catalog-api } }
terminationGracePeriodSeconds: 40
containers:
- name: api
lifecycle: { preStop: { sleep: { seconds: 10 } } } # (3)
readinessProbe: { httpGet: { path: /ready, port: http } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: catalog-api
spec:
maxUnavailable: 1 # (4)
selector: { matchLabels: { app.kubernetes.io/name: catalog-api } }- During a rollout: start a third pod, wait until it is ready, then stop an old one.
- Never both replicas on the same node.
- Keep serving while routing catches up during shutdown (question 488).
- Drains, autoscaler scale-downs and upgrades may take at most one pod at a time.
Why three is often better than two. With two replicas, each must be able to carry all the traffic alone during any disruption. If each normally runs at 60% of its capacity, losing one puts the other at 120%: overloaded exactly when you are changing things. With three, losing one leaves two at 90%. This is the idea of N+1 capacity: enough to lose one and still cope.
430. What baseline controls would you require before allowing a team to deploy workloads into a shared cluster?
The short answer
Each team gets its own namespace, and its access is limited to that namespace. The namespace has resource quotas and default limits, and the restricted Pod Security level. It has a default-deny network policy with explicit allowances. And rules only allow images from approved, signed sources. Every workload must declare resource requests, probes and ownership labels. Secrets come from the outside secret manager, and logging, monitoring and cost reporting are connected from day one. Most of this is applied automatically when the namespace is created.
| Area | Baseline |
|---|---|
| Isolation | One namespace per team or app (question 325) |
| Access | Team group gets edit in its namespaces; nobody gets cluster-admin (question 415) |
| Resources | ResourceQuota and LimitRange in every namespace (question 493) |
| Pod security | Pod Security Admission restricted enforced (question 499) |
| Network | Default-deny policies, plus allowances for DNS and declared dependencies (question 367) |
| Images | Only from approved registries; signed by the release pipeline (question 147) |
| Workload rules | Requests, readiness probes, non-root, no latest tags, enforced by policy (question 500) |
| Ownership | Labels for team, service and cost centre on every object |
| Secrets | From the outside secret manager, through workload identity (question 492) |
| Availability | Guidance on replicas and disruption budgets for production |
| Observability | Logs, metrics and alerts connected automatically |
| Cost | Usage reported per namespace or team label |
Make it self-service. A platform team should not apply this by hand for every new team. A namespace template, applied through GitOps or a small operator, creates the namespace with its quota, limits, policies and role bindings in one step. Policies then keep everything compliant afterwards, and report rather than block at first, so teams can fix things before enforcement starts.
499. What is Pod Security Admission, and what are the privileged, baseline, and restricted levels?
The short answer
Pod Security Admission is a built-in admission controller that checks every pod against the Pod Security Standards, a set of three levels. Privileged allows everything. Baseline blocks the well-known ways to gain extra privileges, such as privileged containers, sharing the host's network or processes, and host folders. Restricted adds hardening on top, such as requiring non-root users, dropping all capabilities and using a system call filter. You choose a level per namespace with labels, in modes that enforce, audit or only warn.
Choosing the level, with namespace labels:
yaml
apiVersion: v1
kind: Namespace
metadata:
name: shop-prod
labels:
pod-security.kubernetes.io/enforce: restricted # (1)
pod-security.kubernetes.io/enforce-version: v1.34 # (2)
pod-security.kubernetes.io/warn: restricted # (3)
pod-security.kubernetes.io/audit: restricted # (4)- Reject pods that do not meet the restricted level.
- Use the rules as defined in that Kubernetes version, so a cluster upgrade does not suddenly change what is allowed.
- Warn the user, for example in
kubectl applyoutput, about violations. - Record violations in the API server's audit log.
The three levels:
| Level | Allows | Typical use |
|---|---|---|
privileged | Everything | System components: network plugins, storage drivers |
baseline | Most things, except known privilege escalations | A minimum for ordinary workloads |
restricted | Only hardened pods | The target for applications |
What each level forbids, in short:
- Baseline forbids privileged containers, the host's network, process and IPC namespaces,
hostPathvolumes, host ports, and adding risky capabilities. - Restricted also requires
runAsNonRoot,allowPrivilegeEscalation: false, dropping all capabilities exceptNET_BIND_SERVICE, a seccomp profile ofRuntimeDefaultorLocalhost, and only ordinary volume types.
What it replaced. An older mechanism, PodSecurityPolicy, was removed in Kubernetes 1.25 because it was confusing to use. Pod Security Admission is simpler, but only offers these three fixed levels. For custom rules, use a policy engine (question 500).
Rolling it out: start with warn and audit at restricted on existing namespaces, fix what they report, then switch on enforce.
500. What are policy engines like Kyverno and OPA Gatekeeper used for?
The short answer
They let you write your own rules for what may run in the cluster, and enforce them at admission, as webhooks the API server calls. Typical rules: images must come from our registry and be signed, every container must set resource requests, no latest tags, every namespace must have an owner label. Kyverno writes policies as ordinary Kubernetes YAML and can also change or generate objects. Gatekeeper uses the Open Policy Agent and its Rego language. Both can audit existing objects and report violations, not only block new ones.
A Kyverno policy:
yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-requests-and-no-latest
spec:
validationFailureAction: Enforce
rules:
- name: require-requests
match: { any: [ { resources: { kinds: [Pod] } } ] }
validate:
message: "Every container must set CPU and memory requests."
pattern:
spec:
containers:
- resources:
requests: { cpu: "?*", memory: "?*" } # (1)
- name: no-latest-tag
match: { any: [ { resources: { kinds: [Pod] } } ] }
validate:
message: "Use a pinned tag or digest, not latest."
pattern:
spec:
containers:
- image: "!*:latest" # (2)?*means "any non-empty value".!means "must not match".
| Kyverno | OPA Gatekeeper | |
|---|---|---|
| Policy language | Kubernetes-style YAML | Rego, a policy language |
| Validate | Yes | Yes |
| Mutate, change objects | Yes | Yes |
| Generate new objects | Yes, for example a default network policy per new namespace | No |
| Verify image signatures | Built in | Through extensions |
| Audit existing objects | Policy reports | Audit results on constraints |
| Learning curve | Gentle for Kubernetes users | Steeper, more expressive |
Built-in alternative. For simple validation, ValidatingAdmissionPolicy with CEL expressions runs inside the API server, with no webhook to operate (question 486).
Rolling out policies safely: start in audit mode, read the reports, fix existing workloads, and only then enforce. Exclude system namespaces, and have an exception process (question 279).
501. How do you upgrade a Kubernetes cluster safely?
The short answer
Prepare, then go one minor version at a time. Read the release notes, and find any APIs your manifests and charts use that the new version removes. Upgrade non-production clusters first. Back up the cluster state. Upgrade the control plane, then the node pools, replacing nodes gradually with drains that respect disruption budgets. Check that add-ons, such as the network plugin, DNS and gateways, support the new version. Watch closely throughout, and have a way back for large jumps, such as a new node pool or a second cluster.
1. Prepare.
- Read the release notes, especially the removed APIs. For example, Kubernetes 1.25 removed old beta versions of several APIs, and manifests still using them fail to apply.
- Scan for deprecated APIs in your manifests, Helm charts and live cluster, with tools such as Pluto or kubent.
- Check add-ons: the network plugin, CoreDNS, the gateway or ingress controller, the CSI drivers, policy engines and operators each support certain Kubernetes versions.
2. Follow the version rules.
- One minor version at a time for the control plane: 1.32 to 1.33 to 1.34, never skipping.
- Control plane first, then nodes. Kubelets may be older than the control plane, up to three minor versions in recent releases, but never newer.
3. Upgrade non-production first, and let it run for a while with real workloads.
4. Back up cluster state, or rely on GitOps plus backups of persistent data (question 502).
5. Upgrade the control plane. On managed clusters, one command or a setting. Running workloads are not touched.
6. Upgrade the nodes, gradually:
- Surge upgrades: add a new node with the new version, cordon and drain an old one, respecting disruption budgets, delete it, repeat (question 426).
- Or blue-green node pools: create a whole new node pool on the new version, move workloads to it, then delete the old pool, which also gives an easy way back.
7. Watch. Error rates, pod restarts, pending pods and events during and after.
8. For risky jumps, such as several versions behind, build a new cluster on the new version and move traffic to it, instead of upgrading in place.
On managed services, use maintenance windows and auto-upgrade channels, which apply patch upgrades automatically and minor upgrades on a schedule you control (question 618).
502. How do you back up and restore a Kubernetes cluster, for example with etcd snapshots or Velero?
The short answer
Back up two different things: the cluster's state, meaning its objects, and the data on its persistent volumes. For self-managed clusters, an etcd snapshot captures all objects at once. For any cluster, Velero backs up chosen objects to object storage and takes snapshots of their volumes, on a schedule, and can restore them into the same or a new cluster. GitOps adds a third copy: Git holds the desired state of everything you deploy. And a backup only counts once you have restored it in practice.
1. etcd snapshots, for self-managed clusters:
bash
ETCDCTL_API=3 etcdctl snapshot save /backups/etcd-2026-10-27.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.keyThis saves every object in the cluster, as it is at that moment. Restoring replaces the whole cluster state with the snapshot's, so it is an all-or-nothing recovery tool. Store snapshots off the machine, and encrypted, since they contain every Secret. On managed clusters, the provider runs etcd, and you use the other methods below.
2. Velero, for any cluster:
bash
velero schedule create shop-nightly --schedule="0 1 * * *" \
--include-namespaces shop-prod --ttl 720h # (1)
velero restore create --from-backup shop-nightly-20261027010000 # (2)- Every night at 01:00, back up the
shop-prodnamespace's objects to object storage, plus snapshots of its persistent volumes, and keep each backup for 30 days. - Restore it, into the same cluster or a new one.
Velero can back up and restore selected namespaces or resources, which makes it useful for recovering one application, for moving workloads to a new cluster, and for disaster recovery.
3. GitOps as a backup of desired state. If every deployment comes from Git, a new cluster can be rebuilt by pointing the GitOps tool at the same repository. What Git cannot restore is data: databases, uploaded files, and objects created at run time.
4. Data needs its own backups. Volume snapshots from Velero or the CSI driver, plus the database's own backups for consistency (question 95).
5. Test restores, regularly, into a scratch cluster, and time them against your recovery objectives (question 307).