Appearance
The API, Objects and Controllers
323. What is a Kubernetes API resource, and how do desired state and observed state differ?
The short answer
A resource is a kind of object the Kubernetes API knows about, such as Pods, Deployments or Services, and an object is one stored instance of it. Most objects have two halves. The spec is the desired state: what you want, written by you. The status is the observed state: what is actually true, written by Kubernetes. Controllers spend all their time making the status match the spec.
Resources are addressed by group, version and kind. A Deployment is kind Deployment in group apps, version v1, so its API address is:
text
/apis/apps/v1/namespaces/shop-prod/deployments/checkout-apiCore resources such as Pods have no group: /api/v1/namespaces/shop-prod/pods.
Spec and status side by side:
bash
kubectl get deployment checkout-api -n shop-prod -o yamlyaml
metadata:
generation: 7 # (1)
spec:
replicas: 3 # (2)
template:
spec:
containers:
- name: api
image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
status:
observedGeneration: 7 # (3)
replicas: 3
readyReplicas: 2 # (4)
updatedReplicas: 3
conditions:
- type: Available
status: "True"
- type: Progressing
status: "True"
reason: ReplicaSetUpdatedgenerationincreases by one every time the spec changes.- Desired: three copies of this image.
- Observed: the controller has processed spec generation 7. If
observedGenerationwere lower thangeneration, the controller would not have caught up with your latest change yet. - Observed: only two of the three copies are ready right now. Something is still starting, or failing.
Why the split matters. You only ever write the spec. Reading the status tells you whether Kubernetes has achieved it. A pipeline that checks only "was my spec accepted?" misses everything the status would have told it (question 230).
324. What is a declarative configuration, and how is it different from imperatively creating resources?
The short answer
Imperative means giving step-by-step commands: create this, scale that to five, change this image. Declarative means describing the end state you want in files, and letting Kubernetes work out what to change to get there, usually with kubectl apply or a GitOps tool. Declarative files can be reviewed, versioned and applied again and again with the same result, which is why production clusters are managed declaratively.
Imperative:
bash
kubectl create deployment checkout-api --image=myacr.azurecr.io/shop/checkout-api:1.4
kubectl scale deployment checkout-api --replicas=5
kubectl set image deployment/checkout-api api=myacr.azurecr.io/shop/checkout-api:1.5Each command is an action. After running them, the only record of what the cluster should look like is the cluster itself and your shell history.
Declarative:
yaml
# deploy/checkout-api.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
namespace: shop-prod
spec:
replicas: 5
selector:
matchLabels: { app: checkout-api }
template:
metadata:
labels: { app: checkout-api }
spec:
containers:
- name: api
image: myacr.azurecr.io/shop/checkout-api:1.5bash
kubectl apply -f deploy/checkout-api.yamlapply compares the file with what is in the cluster and changes only what differs. Run it ten times and the result is the same, which is called idempotent.
| Imperative | Declarative | |
|---|---|---|
| You describe | Actions | The end state |
| Repeatable | No, "create" fails the second time | Yes |
| Reviewable | Not really | Yes, as a pull request |
| History | Shell history | Git history |
| Drift detection | None | Compare files with the cluster |
| Good for | Quick experiments, debugging | Everything that lasts |
A useful habit: use imperative commands to generate declarative files, then edit and commit them:
bash
kubectl create deployment checkout-api --image=nginx --dry-run=client -o yaml > deployment.yaml325. What is a namespace, and when would you use namespaces to separate workloads?
The short answer
A namespace is a named partition inside one cluster. Names of objects must be unique only within a namespace, and namespaces are where you attach permissions, resource quotas, default limits and network policies. Use them to separate teams, applications, or small environments sharing a cluster. They are not a strong security wall on their own, because pods in different namespaces still share the same nodes and kernel.
bash
kubectl create namespace shop-prod
kubectl get pods -n shop-prod
kubectl get pods --all-namespacesWhat a namespace gives you:
- Separate names.
checkout-apican exist in bothshop-devandshop-prod. - Permissions per namespace. A team can be allowed to manage everything in
shop-devand only readshop-prod(question 415). - Quotas and default limits for the namespace as a whole (question 493).
- Network policies that, for example, allow only pods in
shop-prodto reach the production database proxy (question 367). - A DNS name: a Service is reachable as
checkout-api.shop-prod.svc.cluster.local(question 364).
Common ways to divide:
| Pattern | Example | Good for |
|---|---|---|
| Per team | payments, catalog | Many teams sharing a cluster |
| Per application | shop, crm | Several apps per team |
| Per environment | shop-dev, shop-qa | Small setups sharing one non-production cluster |
| System components | kube-system, monitoring, ingress | Separating platform tools |
What namespaces do not separate: nodes, the kernel, and cluster-wide resources. Some objects are cluster-scoped, existing outside any namespace: nodes, persistent volumes, storage classes, cluster roles and namespaces themselves. For production versus non-production, many companies use separate clusters, not just namespaces (question 453).
326. What are labels and selectors, and which Kubernetes resources depend on them?
The short answer
Labels are short key and value pairs attached to objects, such as app: checkout-api or env: prod, used to identify and group them. Selectors are queries over labels, such as "every pod with app: checkout-api". Many resources find the objects they manage this way: Services find their pods, Deployments and ReplicaSets find their pods, network policies choose pods, disruption budgets protect pods, and scheduling rules choose nodes.
yaml
metadata:
labels:
app.kubernetes.io/name: checkout-api # (1)
app.kubernetes.io/part-of: shop
env: prod
team: checkout- Kubernetes recommends a standard set of label keys starting with
app.kubernetes.io/, so tools can understand them.
Two kinds of selector:
bash
kubectl get pods -l app.kubernetes.io/name=checkout-api # equality
kubectl get pods -l 'env in (prod,staging),team!=payments' # set-basedResources that depend on labels:
| Resource | Uses labels to |
|---|---|
| Service | Choose which pods receive traffic (question 360) |
| Deployment, ReplicaSet, StatefulSet, DaemonSet | Know which pods are theirs (question 347) |
| Job | Track its pods |
| NetworkPolicy | Choose which pods a rule applies to, and which pods or namespaces may connect |
| PodDisruptionBudget | Choose which pods it protects (question 400) |
| Pod scheduling rules | Choose nodes by node labels, or place near or away from other pods (questions 394 and 395) |
| Topology spread | Count matching pods per zone (question 396) |
A classic mistake: two different apps sharing a label like app: api. A Service meant for one now sends traffic to both. Use specific, consistent labels.
327. What are annotations, and how are they different from labels?
The short answer
Annotations are also key and value pairs on objects, but they are for information, not for selecting. They hold anything tools or people want to attach, such as build details, configuration for a controller, or notes, and their values can be long. Labels are short, follow strict rules, and are used by selectors to find and group objects. You cannot select objects by annotation.
yaml
metadata:
labels:
app.kubernetes.io/name: checkout-api # (1)
annotations:
shop.example.com/commit: 3f9c2ab8e71d # (2)
shop.example.com/owner: checkout-team
kubernetes.io/change-cause: "deploy 1.4.2 from pipeline 9137" # (3)
prometheus.io/scrape: "true" # (4)- A label, used by Services and Deployments to find these pods.
- Build traceability, read by people and tools (question 196).
- Shown by
kubectl rollout historyas the reason for each revision (question 352). - A hint read by some monitoring setups.
| Labels | Annotations | |
|---|---|---|
| Purpose | Identify and select | Describe and configure |
| Used in selectors | Yes | No |
| Value size | Short, at most 63 characters | Can be large |
| Allowed characters | Restricted | Anything |
| Typical readers | Controllers, Services, kubectl -l | Tools, controllers, people |
Annotations as controller settings. Many controllers read their configuration from annotations, for example an ingress controller reading nginx.ingress.kubernetes.io/proxy-body-size: 10m. This is convenient but has no schema checking, so a typo is silently ignored. Newer APIs, such as Gateway API, move such settings into proper fields (question 370).
328. What is a context in a kubeconfig file, and how do you avoid running commands against the wrong cluster?
The short answer
A context is a named combination of a cluster, a user identity and a default namespace in your kubeconfig file. kubectl uses the current context for every command, so a forgotten switch can send a delete meant for development to production. Protect yourself by showing the current context in your shell prompt, using separate kubeconfig files per environment, passing --context explicitly in scripts, and, most of all, not having routine write access to production at all.
bash
kubectl config get-contexts # list contexts; * marks the current one
kubectl config current-context # which one is active
kubectl config use-context dev # switch
kubectl --context prod get pods -n shop-prod # one command against another contextHabits that prevent accidents:
- Show it in the prompt. Tools such as
kube-ps1add the current context and namespace to your shell prompt, and you can colour production red. - Separate files per environment. Keep production credentials in their own file, and point at it only when needed with
KUBECONFIG=~/.kube/prod kubectl .... By default, your shell knows only non-production clusters. - Explicit context in scripts. Every script states
--contextand--namespace. Never rely on whatever happens to be current. - Helpers.
kubectxandkubensswitch contexts and namespaces quickly and visibly. - Read-only by default in production. Day-to-day credentials can only read. Changing production needs a separate, elevated, time-limited role (question 271).
- GitOps for production changes, so
kubectl applyagainst production is not part of normal work at all (question 507).
The real lesson: good habits reduce mistakes, but permissions remove them. If your normal identity cannot delete production namespaces, the wrong context cannot either.
329. How do kubectl get, describe, logs, exec, apply, diff, and delete differ?
The short answer
get lists objects and their basic state. describe shows one object in detail, including its recent events, which is where most problems are explained. logs prints a container's output. exec runs a command inside a running container. apply creates or updates objects from files to match them. diff shows what apply would change before you run it. delete removes objects, gracefully by default.
| Command | Use it to | Example |
|---|---|---|
get | List and filter objects | kubectl get pods -n shop-prod -o wide |
describe | Details and events of one object | kubectl describe pod checkout-api-7d9f-abcde |
logs | Read container output | kubectl logs deploy/checkout-api --previous |
exec | Run a command inside a container | kubectl exec -it checkout-api-7d9f-abcde -- sh |
apply | Create or update from files | kubectl apply -f deploy/ |
diff | Preview what apply would change | kubectl diff -f deploy/ |
delete | Remove objects | kubectl delete pod checkout-api-7d9f-abcde |
Details worth knowing:
get -o wideadds the node and the pod IP.-wwatches for changes live.-o yamlshows the full object, spec and status (question 323).describe's Events section, at the bottom, lists what happened recently: scheduling decisions, image pulls, probe failures, kills. It is the first place to look for almost any pod problem (question 410).logs --previousshows the output of the container's last run, before it restarted. When a container crash-loops, the current run's log is often empty, and this is the one you need.-cpicks a container in a multi-container pod, and-ffollows new output.execonly works if the container has the command, such assh. For images without a shell, use an ephemeral debug container (question 414).applyrecords what you applied, so later applies can tell your fields apart from fields set by others (question 348).diffasks the API server to work out the change without saving it, so admission defaults and webhooks are included in the preview.deletesends the pod a stop signal and waits for its grace period. Deleting a pod owned by a Deployment just makes the Deployment create a replacement, which is a quick way to restart one pod.
330. What is a Custom Resource Definition (CRD), and what does it add to the Kubernetes API?
The short answer
A Custom Resource Definition teaches the Kubernetes API a new kind of object. After you create one, the API server stores and serves the new kind exactly like built-in ones: you can create objects of that kind with kubectl apply, list them, watch them, and control access with RBAC. A CRD on its own only stores data. Something must act on that data, which is the job of a controller, and together they are called an operator.
A CRD, simplified:
yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.shop.example.com
spec:
group: shop.example.com
names: { kind: Database, plural: databases, singular: database }
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [engine, sizeGb]
properties:
engine: { type: string, enum: [postgres, mysql] }
sizeGb: { type: integer, minimum: 10 }An object of the new kind:
yaml
apiVersion: shop.example.com/v1
kind: Database
metadata:
name: orders
namespace: shop-prod
spec:
engine: postgres
sizeGb: 100bash
kubectl get databases -n shop-prodWhat the CRD gives you:
- A new API type with schema checking:
sizeGb: 5is rejected, because the minimum is 10. - All the normal tooling:
kubectl, RBAC, watches, labels, GitOps. - No behaviour. Nothing creates a database yet. A controller must watch
Databaseobjects and do the work (question 331).
Real examples you will meet: cert-manager's Certificate, Argo CD's Application, Argo Rollouts' Rollout, the Prometheus operator's ServiceMonitor, and Gateway API's HTTPRoute, which started as CRDs and became a standard Kubernetes API that way.
331. What is an operator, and how does it use custom resources and controllers?
The short answer
An operator is a custom controller, usually paired with its own custom resources, that encodes the knowledge of how to run one particular piece of software: install it, configure it, upgrade it, back it up, recover it. You describe what you want in a custom resource, such as "a PostgreSQL cluster with three instances and daily backups", and the operator keeps making it so, doing the work a human administrator would otherwise do by hand.
An example, using the CloudNativePG operator for PostgreSQL:
yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: orders-db
namespace: shop-prod
spec:
instances: 3
storage: { size: 100Gi }
backup:
barmanObjectStore:
destinationPath: s3://shop-backups/orders-dbFrom this one object, the operator:
- Creates three PostgreSQL pods, one primary and two replicas, with their storage.
- Sets up replication between them.
- Creates Services: one that always points at the current primary, one for the replicas.
- Takes backups to object storage on schedule.
- Fails over: if the primary dies, it promotes a replica and repoints the primary Service.
- Handles minor upgrades, one instance at a time.
How it works inside. The operator is a controller (question 332) watching its custom resources, and the pods, services and other objects it created. Each time anything changes, it compares the desired state from the custom resource with reality, and acts.
Common operators: the Prometheus operator, cert-manager for TLS certificates, Strimzi for Kafka, and operators for most databases. Tools such as Kubebuilder and the Operator SDK help teams write their own.
A word of caution in interviews: an operator makes running complex software easier, not trivial. You still need to understand what it does when something fails, because when it misbehaves, you are the one debugging it (question 169).
332. How do Kubernetes controllers reconcile actual state toward desired state?
The short answer
Each controller runs a loop. It watches the objects it cares about through the API server. Whenever something changes, and on a regular schedule anyway, it reads the desired state and observes the actual state. It works out the difference, and takes one step to reduce it, such as creating a missing pod. Then it records what it saw in the object's status, and repeats. Because it always re-reads the whole state, it fixes problems even if it missed an event.
The loop in detail:
- Observe. Controllers keep a local, constantly updated copy of the objects they care about, fed by watches. This copy is often called an informer cache. Reading from it is fast and does not load the API server.
- Compare. Desired: the ReplicaSet's
spec.replicas: 3. Actual: two matching pods exist. - Act. Create one pod through the API server. One step at a time, not a big plan.
- Record. Write what was observed into the object's
status, such asreadyReplicas: 2. - Repeat. The new pod's creation is itself a change, which triggers the loop again.
Level-triggered, not edge-triggered. This is the key design idea, and interviewers love it.
- An edge-triggered system reacts to events: "a pod was deleted, so create one". If it misses the event, for example because it was restarting at that moment, the problem is never fixed.
- A level-triggered system reacts to state: "there should be 3, there are 2, so create one". It does not matter how or when the state became wrong. Every pass looks at the whole picture.
Other properties that make it robust:
- Idempotent actions: doing the same step twice causes no harm.
- Retries with backoff: a failed action is retried later, with growing delays.
- Periodic resync: even without events, controllers re-check everything from time to time.
- Ownership: controllers only manage objects they own, recorded in owner references (question 487).
333. What happens at a high level from submitting a Deployment manifest to a new Pod becoming Ready?
The short answer
The API server checks and stores the Deployment. The Deployment controller creates a ReplicaSet, and the ReplicaSet controller creates pods. The scheduler assigns each pod to a node. That node's kubelet asks the container runtime to pull the image, set up the pod's network and storage, and start its containers. Probes then confirm the app is ready, the kubelet marks the pod Ready, and the pod's address is added to its Service's endpoints, so traffic starts flowing to it.
The steps, in plain words:
kubectl applysends the Deployment to the API server.- The API server authenticates you, checks your permission, runs admission webhooks and policies, validates the object, and stores it in etcd (question 314).
- The Deployment controller sees a new Deployment and creates a ReplicaSet for its pod template (question 338).
- The ReplicaSet controller sees it needs three pods and none exist, so it creates three pod objects. They have no node yet, and their phase is
Pending. - The scheduler sees pods without a node, filters and scores the nodes, and binds each pod to one (question 316).
- The kubelet on each chosen node sees a pod assigned to it.
- The container runtime pulls the image, if the node does not have it, and creates the pod's sandbox: its network namespace, with an IP address given by the network plugin (question 366).
- Init containers run to completion, in order, then the app containers start (question 355).
- Probes: the startup probe must pass first, then the readiness probe. Only then does the kubelet set the pod's
Readycondition to true (question 401). - The EndpointSlice controller sees a ready pod matching the Service's selector, and adds its IP address to the Service's endpoints (question 361).
- kube-proxy, or the network plugin, updates the rules on every node, and requests to the Service now reach the new pod.
Where it most often gets stuck, and the question that covers each. Step 2 can be rejected by a policy (question 486). At step 5, no node may fit (question 408). At step 7, the image may not pull (question 406). At step 8, the container may crash (question 407). At step 9, it may never become ready (question 409).
486. What are admission controllers and admission webhooks in Kubernetes?
The short answer
Admission controllers are checks inside the API server that run on every request to create or change an object, after the requester is identified and authorised, but before anything is stored. Mutating admission can change the object, for example adding defaults or injecting a sidecar container. Validating admission can only accept or reject it, for example "images must come from our registry". Webhooks let outside services, such as Kyverno or Gatekeeper, take part. Newer clusters can also write validation rules directly in the API server, without a webhook.
Built-in admission controllers are compiled into the API server and switched on by default in most clusters. Examples:
| Controller | What it does |
|---|---|
NamespaceLifecycle | Refuses new objects in a namespace being deleted |
LimitRanger | Applies default resource requests and limits (question 493) |
ResourceQuota | Rejects objects that would exceed a namespace's quota |
ServiceAccount | Gives pods a default service account |
DefaultStorageClass | Fills in the default storage class on volume claims |
PodSecurity | Enforces Pod Security Standards (question 499) |
Admission webhooks call outside services. You register them with MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects, which say which requests to send where. Policy engines such as Kyverno and Gatekeeper work this way (question 500), and so do service meshes that inject sidecars (question 515).
Writing rules without a webhook. ValidatingAdmissionPolicy, stable since Kubernetes 1.30, lets you write validation rules in CEL, a small expression language, directly in the API server:
yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: images-from-our-registry
spec:
matchConstraints:
resourceRules:
- { apiGroups: ["apps"], apiVersions: ["v1"], operations: ["CREATE", "UPDATE"], resources: ["deployments"] }
validations:
- expression: "object.spec.template.spec.containers.all(c, c.image.startsWith('myacr.azurecr.io/'))"
message: "images must come from myacr.azurecr.io"A separate ValidatingAdmissionPolicyBinding decides which namespaces the policy applies to.
The operational trap. A webhook with failurePolicy: Fail blocks requests whenever the webhook is down. If the webhook service itself runs in the cluster, and it crashes, you may be unable to create the very pods that would fix it. Run webhooks with several replicas, exclude system namespaces from them, and know how to disable a broken one.
487. What are finalizers and owner references, and why do objects sometimes get stuck in Terminating?
The short answer
An owner reference records that one object belongs to another, for example a pod belongs to a ReplicaSet, which belongs to a Deployment. When an owner is deleted, the garbage collector deletes what it owned. A finalizer is a marker on an object that says "do not finish deleting me until a certain clean-up has been done". The controller responsible does the clean-up, then removes its marker. An object stays in Terminating when a finalizer is never removed, usually because that controller is broken, missing, or failing at its clean-up.
Owner references, in a pod's metadata:
yaml
metadata:
name: checkout-api-7d9f-abcde
ownerReferences:
- apiVersion: apps/v1
kind: ReplicaSet
name: checkout-api-7d9f
controller: trueDeleting the Deployment deletes its ReplicaSets, which deletes their pods. This is cascading deletion, done by the garbage collector controller. kubectl delete can also orphan the dependents instead, with --cascade=orphan, leaving them running without an owner.
Finalizers, in an object's metadata:
yaml
metadata:
name: orders-data
finalizers:
- kubernetes.io/pvc-protection
deletionTimestamp: "2026-10-27T09:14:02Z"When you delete an object that has finalizers, Kubernetes does not remove it. It sets deletionTimestamp, and the object shows as Terminating. Each controller named by a finalizer does its clean-up and removes its entry. When the list is empty, the object is really deleted.
Common finalizers:
| Finalizer | Protects against |
|---|---|
kubernetes.io/pvc-protection | Deleting a volume claim while a pod still uses it |
kubernetes.io/pv-protection | Deleting a volume still bound to a claim |
service.kubernetes.io/load-balancer-cleanup | Leaving a cloud load balancer behind when a Service is deleted |
| Namespace finalizers | Deleting a namespace before everything inside it is gone |
| Operators' own finalizers | Deleting a custom resource before the operator removes the real database or bucket |
Stuck in Terminating: how to investigate.
- Look at the finalizers:
kubectl get pvc orders-data -o jsonpath='{.metadata.finalizers}'. - Find the responsible controller. Is it running? Do its logs show an error, such as a cloud permission failure while deleting a load balancer?
- Fix the cause, so the controller can finish its clean-up.
- Only as a last resort, remove the finalizer by hand, with
kubectl patch ... --type=merge -p '{"metadata":{"finalizers":null}}'. This skips the clean-up, so you may leave a real cloud resource, or data, behind. Know what you are skipping before you do it.
A namespace stuck in Terminating usually contains an object whose finalizer cannot be processed, often a custom resource whose operator was deleted first. kubectl get namespace shop-old -o yaml shows conditions listing what is blocking it.