Skip to content

Interview PrepKubernetes

The API, Objects and Controllers ​

323. What is a Kubernetes API resource, and how do desired state and observed state differ? ​

The short answer

A resource is a kind of object the Kubernetes API knows about, such as Pods, Deployments or Services, and an object is one stored instance of it. Most objects have two halves. The spec is the desired state: what you want, written by you. The status is the observed state: what is actually true, written by Kubernetes. Controllers spend all their time making the status match the spec.

Resources are addressed by group, version and kind. A Deployment is kind Deployment in group apps, version v1, so its API address is:

text
/apis/apps/v1/namespaces/shop-prod/deployments/checkout-api

Core resources such as Pods have no group: /api/v1/namespaces/shop-prod/pods.

Spec and status side by side:

bash
kubectl get deployment checkout-api -n shop-prod -o yaml
yaml
metadata:
  generation: 7                  # (1)
spec:
  replicas: 3                    # (2)
  template:
    spec:
      containers:
        - name: api
          image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
status:
  observedGeneration: 7          # (3)
  replicas: 3
  readyReplicas: 2               # (4)
  updatedReplicas: 3
  conditions:
    - type: Available
      status: "True"
    - type: Progressing
      status: "True"
      reason: ReplicaSetUpdated
  1. generation increases by one every time the spec changes.
  2. Desired: three copies of this image.
  3. Observed: the controller has processed spec generation 7. If observedGeneration were lower than generation, the controller would not have caught up with your latest change yet.
  4. Observed: only two of the three copies are ready right now. Something is still starting, or failing.

Why the split matters. You only ever write the spec. Reading the status tells you whether Kubernetes has achieved it. A pipeline that checks only "was my spec accepted?" misses everything the status would have told it (question 230).

324. What is a declarative configuration, and how is it different from imperatively creating resources? ​

The short answer

Imperative means giving step-by-step commands: create this, scale that to five, change this image. Declarative means describing the end state you want in files, and letting Kubernetes work out what to change to get there, usually with kubectl apply or a GitOps tool. Declarative files can be reviewed, versioned and applied again and again with the same result, which is why production clusters are managed declaratively.

Imperative:

bash
kubectl create deployment checkout-api --image=myacr.azurecr.io/shop/checkout-api:1.4
kubectl scale deployment checkout-api --replicas=5
kubectl set image deployment/checkout-api api=myacr.azurecr.io/shop/checkout-api:1.5

Each command is an action. After running them, the only record of what the cluster should look like is the cluster itself and your shell history.

Declarative:

yaml
# deploy/checkout-api.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: shop-prod
spec:
  replicas: 5
  selector:
    matchLabels: { app: checkout-api }
  template:
    metadata:
      labels: { app: checkout-api }
    spec:
      containers:
        - name: api
          image: myacr.azurecr.io/shop/checkout-api:1.5
bash
kubectl apply -f deploy/checkout-api.yaml

apply compares the file with what is in the cluster and changes only what differs. Run it ten times and the result is the same, which is called idempotent.

ImperativeDeclarative
You describeActionsThe end state
RepeatableNo, "create" fails the second timeYes
ReviewableNot reallyYes, as a pull request
HistoryShell historyGit history
Drift detectionNoneCompare files with the cluster
Good forQuick experiments, debuggingEverything that lasts

A useful habit: use imperative commands to generate declarative files, then edit and commit them:

bash
kubectl create deployment checkout-api --image=nginx --dry-run=client -o yaml > deployment.yaml

325. What is a namespace, and when would you use namespaces to separate workloads? ​

The short answer

A namespace is a named partition inside one cluster. Names of objects must be unique only within a namespace, and namespaces are where you attach permissions, resource quotas, default limits and network policies. Use them to separate teams, applications, or small environments sharing a cluster. They are not a strong security wall on their own, because pods in different namespaces still share the same nodes and kernel.

bash
kubectl create namespace shop-prod
kubectl get pods -n shop-prod
kubectl get pods --all-namespaces

What a namespace gives you:

  1. Separate names. checkout-api can exist in both shop-dev and shop-prod.
  2. Permissions per namespace. A team can be allowed to manage everything in shop-dev and only read shop-prod (question 415).
  3. Quotas and default limits for the namespace as a whole (question 493).
  4. Network policies that, for example, allow only pods in shop-prod to reach the production database proxy (question 367).
  5. A DNS name: a Service is reachable as checkout-api.shop-prod.svc.cluster.local (question 364).

Common ways to divide:

PatternExampleGood for
Per teampayments, catalogMany teams sharing a cluster
Per applicationshop, crmSeveral apps per team
Per environmentshop-dev, shop-qaSmall setups sharing one non-production cluster
System componentskube-system, monitoring, ingressSeparating platform tools

What namespaces do not separate: nodes, the kernel, and cluster-wide resources. Some objects are cluster-scoped, existing outside any namespace: nodes, persistent volumes, storage classes, cluster roles and namespaces themselves. For production versus non-production, many companies use separate clusters, not just namespaces (question 453).

326. What are labels and selectors, and which Kubernetes resources depend on them? ​

The short answer

Labels are short key and value pairs attached to objects, such as app: checkout-api or env: prod, used to identify and group them. Selectors are queries over labels, such as "every pod with app: checkout-api". Many resources find the objects they manage this way: Services find their pods, Deployments and ReplicaSets find their pods, network policies choose pods, disruption budgets protect pods, and scheduling rules choose nodes.

yaml
metadata:
  labels:
    app.kubernetes.io/name: checkout-api      # (1)
    app.kubernetes.io/part-of: shop
    env: prod
    team: checkout
  1. Kubernetes recommends a standard set of label keys starting with app.kubernetes.io/, so tools can understand them.

Two kinds of selector:

bash
kubectl get pods -l app.kubernetes.io/name=checkout-api            # equality
kubectl get pods -l 'env in (prod,staging),team!=payments'          # set-based

Resources that depend on labels:

ResourceUses labels to
ServiceChoose which pods receive traffic (question 360)
Deployment, ReplicaSet, StatefulSet, DaemonSetKnow which pods are theirs (question 347)
JobTrack its pods
NetworkPolicyChoose which pods a rule applies to, and which pods or namespaces may connect
PodDisruptionBudgetChoose which pods it protects (question 400)
Pod scheduling rulesChoose nodes by node labels, or place near or away from other pods (questions 394 and 395)
Topology spreadCount matching pods per zone (question 396)

A classic mistake: two different apps sharing a label like app: api. A Service meant for one now sends traffic to both. Use specific, consistent labels.

327. What are annotations, and how are they different from labels? ​

The short answer

Annotations are also key and value pairs on objects, but they are for information, not for selecting. They hold anything tools or people want to attach, such as build details, configuration for a controller, or notes, and their values can be long. Labels are short, follow strict rules, and are used by selectors to find and group objects. You cannot select objects by annotation.

yaml
metadata:
  labels:
    app.kubernetes.io/name: checkout-api                            # (1)
  annotations:
    shop.example.com/commit: 3f9c2ab8e71d                           # (2)
    shop.example.com/owner: checkout-team
    kubernetes.io/change-cause: "deploy 1.4.2 from pipeline 9137"   # (3)
    prometheus.io/scrape: "true"                                    # (4)
  1. A label, used by Services and Deployments to find these pods.
  2. Build traceability, read by people and tools (question 196).
  3. Shown by kubectl rollout history as the reason for each revision (question 352).
  4. A hint read by some monitoring setups.
LabelsAnnotations
PurposeIdentify and selectDescribe and configure
Used in selectorsYesNo
Value sizeShort, at most 63 charactersCan be large
Allowed charactersRestrictedAnything
Typical readersControllers, Services, kubectl -lTools, controllers, people

Annotations as controller settings. Many controllers read their configuration from annotations, for example an ingress controller reading nginx.ingress.kubernetes.io/proxy-body-size: 10m. This is convenient but has no schema checking, so a typo is silently ignored. Newer APIs, such as Gateway API, move such settings into proper fields (question 370).

328. What is a context in a kubeconfig file, and how do you avoid running commands against the wrong cluster? ​

The short answer

A context is a named combination of a cluster, a user identity and a default namespace in your kubeconfig file. kubectl uses the current context for every command, so a forgotten switch can send a delete meant for development to production. Protect yourself by showing the current context in your shell prompt, using separate kubeconfig files per environment, passing --context explicitly in scripts, and, most of all, not having routine write access to production at all.

bash
kubectl config get-contexts                # list contexts; * marks the current one
kubectl config current-context             # which one is active
kubectl config use-context dev             # switch
kubectl --context prod get pods -n shop-prod   # one command against another context

Habits that prevent accidents:

  1. Show it in the prompt. Tools such as kube-ps1 add the current context and namespace to your shell prompt, and you can colour production red.
  2. Separate files per environment. Keep production credentials in their own file, and point at it only when needed with KUBECONFIG=~/.kube/prod kubectl .... By default, your shell knows only non-production clusters.
  3. Explicit context in scripts. Every script states --context and --namespace. Never rely on whatever happens to be current.
  4. Helpers. kubectx and kubens switch contexts and namespaces quickly and visibly.
  5. Read-only by default in production. Day-to-day credentials can only read. Changing production needs a separate, elevated, time-limited role (question 271).
  6. GitOps for production changes, so kubectl apply against production is not part of normal work at all (question 507).

The real lesson: good habits reduce mistakes, but permissions remove them. If your normal identity cannot delete production namespaces, the wrong context cannot either.

329. How do kubectl get, describe, logs, exec, apply, diff, and delete differ? ​

The short answer

get lists objects and their basic state. describe shows one object in detail, including its recent events, which is where most problems are explained. logs prints a container's output. exec runs a command inside a running container. apply creates or updates objects from files to match them. diff shows what apply would change before you run it. delete removes objects, gracefully by default.

CommandUse it toExample
getList and filter objectskubectl get pods -n shop-prod -o wide
describeDetails and events of one objectkubectl describe pod checkout-api-7d9f-abcde
logsRead container outputkubectl logs deploy/checkout-api --previous
execRun a command inside a containerkubectl exec -it checkout-api-7d9f-abcde -- sh
applyCreate or update from fileskubectl apply -f deploy/
diffPreview what apply would changekubectl diff -f deploy/
deleteRemove objectskubectl delete pod checkout-api-7d9f-abcde

Details worth knowing:

  • get -o wide adds the node and the pod IP. -w watches for changes live. -o yaml shows the full object, spec and status (question 323).
  • describe's Events section, at the bottom, lists what happened recently: scheduling decisions, image pulls, probe failures, kills. It is the first place to look for almost any pod problem (question 410).
  • logs --previous shows the output of the container's last run, before it restarted. When a container crash-loops, the current run's log is often empty, and this is the one you need. -c picks a container in a multi-container pod, and -f follows new output.
  • exec only works if the container has the command, such as sh. For images without a shell, use an ephemeral debug container (question 414).
  • apply records what you applied, so later applies can tell your fields apart from fields set by others (question 348).
  • diff asks the API server to work out the change without saving it, so admission defaults and webhooks are included in the preview.
  • delete sends the pod a stop signal and waits for its grace period. Deleting a pod owned by a Deployment just makes the Deployment create a replacement, which is a quick way to restart one pod.

330. What is a Custom Resource Definition (CRD), and what does it add to the Kubernetes API? ​

The short answer

A Custom Resource Definition teaches the Kubernetes API a new kind of object. After you create one, the API server stores and serves the new kind exactly like built-in ones: you can create objects of that kind with kubectl apply, list them, watch them, and control access with RBAC. A CRD on its own only stores data. Something must act on that data, which is the job of a controller, and together they are called an operator.

A CRD, simplified:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.shop.example.com
spec:
  group: shop.example.com
  names: { kind: Database, plural: databases, singular: database }
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [engine, sizeGb]
              properties:
                engine: { type: string, enum: [postgres, mysql] }
                sizeGb: { type: integer, minimum: 10 }

An object of the new kind:

yaml
apiVersion: shop.example.com/v1
kind: Database
metadata:
  name: orders
  namespace: shop-prod
spec:
  engine: postgres
  sizeGb: 100
bash
kubectl get databases -n shop-prod

What the CRD gives you:

  1. A new API type with schema checking: sizeGb: 5 is rejected, because the minimum is 10.
  2. All the normal tooling: kubectl, RBAC, watches, labels, GitOps.
  3. No behaviour. Nothing creates a database yet. A controller must watch Database objects and do the work (question 331).

Real examples you will meet: cert-manager's Certificate, Argo CD's Application, Argo Rollouts' Rollout, the Prometheus operator's ServiceMonitor, and Gateway API's HTTPRoute, which started as CRDs and became a standard Kubernetes API that way.

331. What is an operator, and how does it use custom resources and controllers? ​

The short answer

An operator is a custom controller, usually paired with its own custom resources, that encodes the knowledge of how to run one particular piece of software: install it, configure it, upgrade it, back it up, recover it. You describe what you want in a custom resource, such as "a PostgreSQL cluster with three instances and daily backups", and the operator keeps making it so, doing the work a human administrator would otherwise do by hand.

An example, using the CloudNativePG operator for PostgreSQL:

yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: orders-db
  namespace: shop-prod
spec:
  instances: 3
  storage: { size: 100Gi }
  backup:
    barmanObjectStore:
      destinationPath: s3://shop-backups/orders-db

From this one object, the operator:

  1. Creates three PostgreSQL pods, one primary and two replicas, with their storage.
  2. Sets up replication between them.
  3. Creates Services: one that always points at the current primary, one for the replicas.
  4. Takes backups to object storage on schedule.
  5. Fails over: if the primary dies, it promotes a replica and repoints the primary Service.
  6. Handles minor upgrades, one instance at a time.

How it works inside. The operator is a controller (question 332) watching its custom resources, and the pods, services and other objects it created. Each time anything changes, it compares the desired state from the custom resource with reality, and acts.

Common operators: the Prometheus operator, cert-manager for TLS certificates, Strimzi for Kafka, and operators for most databases. Tools such as Kubebuilder and the Operator SDK help teams write their own.

A word of caution in interviews: an operator makes running complex software easier, not trivial. You still need to understand what it does when something fails, because when it misbehaves, you are the one debugging it (question 169).

332. How do Kubernetes controllers reconcile actual state toward desired state? ​

The short answer

Each controller runs a loop. It watches the objects it cares about through the API server. Whenever something changes, and on a regular schedule anyway, it reads the desired state and observes the actual state. It works out the difference, and takes one step to reduce it, such as creating a missing pod. Then it records what it saw in the object's status, and repeats. Because it always re-reads the whole state, it fixes problems even if it missed an event.

① observedesired and actual② comparefind the difference③ actcreate, update, delete④ recordupdate statusexample:want 3 podssee 2 podsdifference 1create 1
The reconciliation loop. Every controller in Kubernetes follows this same shape, from the ReplicaSet controller to a database operator. It never asks "what event happened?", only "what is different, and what is the next step?".

The loop in detail:

  1. Observe. Controllers keep a local, constantly updated copy of the objects they care about, fed by watches. This copy is often called an informer cache. Reading from it is fast and does not load the API server.
  2. Compare. Desired: the ReplicaSet's spec.replicas: 3. Actual: two matching pods exist.
  3. Act. Create one pod through the API server. One step at a time, not a big plan.
  4. Record. Write what was observed into the object's status, such as readyReplicas: 2.
  5. Repeat. The new pod's creation is itself a change, which triggers the loop again.

Level-triggered, not edge-triggered. This is the key design idea, and interviewers love it.

  • An edge-triggered system reacts to events: "a pod was deleted, so create one". If it misses the event, for example because it was restarting at that moment, the problem is never fixed.
  • A level-triggered system reacts to state: "there should be 3, there are 2, so create one". It does not matter how or when the state became wrong. Every pass looks at the whole picture.

Other properties that make it robust:

  • Idempotent actions: doing the same step twice causes no harm.
  • Retries with backoff: a failed action is retried later, with growing delays.
  • Periodic resync: even without events, controllers re-check everything from time to time.
  • Ownership: controllers only manage objects they own, recorded in owner references (question 487).

333. What happens at a high level from submitting a Deployment manifest to a new Pod becoming Ready? ​

The short answer

The API server checks and stores the Deployment. The Deployment controller creates a ReplicaSet, and the ReplicaSet controller creates pods. The scheduler assigns each pod to a node. That node's kubelet asks the container runtime to pull the image, set up the pod's network and storage, and start its containers. Probes then confirm the app is ready, the kubelet marks the pod Ready, and the pod's address is added to its Service's endpoints, so traffic starts flowing to it.

control plane① kubectl apply sends the Deployment② API server: authenticate, authorise, admission, store in etcd③ Deployment controller creates a ReplicaSet④ ReplicaSet controller creates pods, with no node yet⑤ scheduler filters and scores nodes, binds each podevery step is a write to the API server, watched by the nextthe chosen node⑥ kubelet sees a pod assigned to it⑦ runtime pulls the image; network plugin gives the pod an IP⑧ init containers run, then the app containers startreadiness⑨ startup and readinessprobes pass; kubeletmarks the pod Readytraffic⑩ EndpointSlice controlleradds the pod's IP to theService's endpoints⑪ kube-proxy, or the networkplugin, updates every node'srules: traffic reaches the podThe pod gets traffic only after step ⑨. Until then, its Service ignores it.
From a YAML file to a pod serving traffic. No component calls the next one. Each writes to the API server, and the next one, watching, picks it up. Read the boxes in the order of the numbers.

The steps, in plain words:

  1. kubectl apply sends the Deployment to the API server.
  2. The API server authenticates you, checks your permission, runs admission webhooks and policies, validates the object, and stores it in etcd (question 314).
  3. The Deployment controller sees a new Deployment and creates a ReplicaSet for its pod template (question 338).
  4. The ReplicaSet controller sees it needs three pods and none exist, so it creates three pod objects. They have no node yet, and their phase is Pending.
  5. The scheduler sees pods without a node, filters and scores the nodes, and binds each pod to one (question 316).
  6. The kubelet on each chosen node sees a pod assigned to it.
  7. The container runtime pulls the image, if the node does not have it, and creates the pod's sandbox: its network namespace, with an IP address given by the network plugin (question 366).
  8. Init containers run to completion, in order, then the app containers start (question 355).
  9. Probes: the startup probe must pass first, then the readiness probe. Only then does the kubelet set the pod's Ready condition to true (question 401).
  10. The EndpointSlice controller sees a ready pod matching the Service's selector, and adds its IP address to the Service's endpoints (question 361).
  11. kube-proxy, or the network plugin, updates the rules on every node, and requests to the Service now reach the new pod.

Where it most often gets stuck, and the question that covers each. Step 2 can be rejected by a policy (question 486). At step 5, no node may fit (question 408). At step 7, the image may not pull (question 406). At step 8, the container may crash (question 407). At step 9, it may never become ready (question 409).

486. What are admission controllers and admission webhooks in Kubernetes? ​

The short answer

Admission controllers are checks inside the API server that run on every request to create or change an object, after the requester is identified and authorised, but before anything is stored. Mutating admission can change the object, for example adding defaults or injecting a sidecar container. Validating admission can only accept or reject it, for example "images must come from our registry". Webhooks let outside services, such as Kyverno or Gatekeeper, take part. Newer clusters can also write validation rules directly in the API server, without a webhook.

authenticatewho?authoriseallowed?mutatingmay change itvalidateschemavalidatingmay reject itetcdwebhooks: Kyverno, Gatekeeper, sidecar injectors
Where admission sits. Every create or update passes through both admission steps. A rejection at either one means nothing is stored, and the requester gets the reason.

Built-in admission controllers are compiled into the API server and switched on by default in most clusters. Examples:

ControllerWhat it does
NamespaceLifecycleRefuses new objects in a namespace being deleted
LimitRangerApplies default resource requests and limits (question 493)
ResourceQuotaRejects objects that would exceed a namespace's quota
ServiceAccountGives pods a default service account
DefaultStorageClassFills in the default storage class on volume claims
PodSecurityEnforces Pod Security Standards (question 499)

Admission webhooks call outside services. You register them with MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects, which say which requests to send where. Policy engines such as Kyverno and Gatekeeper work this way (question 500), and so do service meshes that inject sidecars (question 515).

Writing rules without a webhook. ValidatingAdmissionPolicy, stable since Kubernetes 1.30, lets you write validation rules in CEL, a small expression language, directly in the API server:

yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: images-from-our-registry
spec:
  matchConstraints:
    resourceRules:
      - { apiGroups: ["apps"], apiVersions: ["v1"], operations: ["CREATE", "UPDATE"], resources: ["deployments"] }
  validations:
    - expression: "object.spec.template.spec.containers.all(c, c.image.startsWith('myacr.azurecr.io/'))"
      message: "images must come from myacr.azurecr.io"

A separate ValidatingAdmissionPolicyBinding decides which namespaces the policy applies to.

The operational trap. A webhook with failurePolicy: Fail blocks requests whenever the webhook is down. If the webhook service itself runs in the cluster, and it crashes, you may be unable to create the very pods that would fix it. Run webhooks with several replicas, exclude system namespaces from them, and know how to disable a broken one.

487. What are finalizers and owner references, and why do objects sometimes get stuck in Terminating? ​

The short answer

An owner reference records that one object belongs to another, for example a pod belongs to a ReplicaSet, which belongs to a Deployment. When an owner is deleted, the garbage collector deletes what it owned. A finalizer is a marker on an object that says "do not finish deleting me until a certain clean-up has been done". The controller responsible does the clean-up, then removes its marker. An object stays in Terminating when a finalizer is never removed, usually because that controller is broken, missing, or failing at its clean-up.

Owner references, in a pod's metadata:

yaml
metadata:
  name: checkout-api-7d9f-abcde
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: checkout-api-7d9f
      controller: true

Deleting the Deployment deletes its ReplicaSets, which deletes their pods. This is cascading deletion, done by the garbage collector controller. kubectl delete can also orphan the dependents instead, with --cascade=orphan, leaving them running without an owner.

Finalizers, in an object's metadata:

yaml
metadata:
  name: orders-data
  finalizers:
    - kubernetes.io/pvc-protection
  deletionTimestamp: "2026-10-27T09:14:02Z"

When you delete an object that has finalizers, Kubernetes does not remove it. It sets deletionTimestamp, and the object shows as Terminating. Each controller named by a finalizer does its clean-up and removes its entry. When the list is empty, the object is really deleted.

Common finalizers:

FinalizerProtects against
kubernetes.io/pvc-protectionDeleting a volume claim while a pod still uses it
kubernetes.io/pv-protectionDeleting a volume still bound to a claim
service.kubernetes.io/load-balancer-cleanupLeaving a cloud load balancer behind when a Service is deleted
Namespace finalizersDeleting a namespace before everything inside it is gone
Operators' own finalizersDeleting a custom resource before the operator removes the real database or bucket

Stuck in Terminating: how to investigate.

  1. Look at the finalizers: kubectl get pvc orders-data -o jsonpath='{.metadata.finalizers}'.
  2. Find the responsible controller. Is it running? Do its logs show an error, such as a cloud permission failure while deleting a load balancer?
  3. Fix the cause, so the controller can finish its clean-up.
  4. Only as a last resort, remove the finalizer by hand, with kubectl patch ... --type=merge -p '{"metadata":{"finalizers":null}}'. This skips the clean-up, so you may leave a real cloud resource, or data, behind. Know what you are skipping before you do it.

A namespace stuck in Terminating usually contains an object whose finalizer cannot be processed, often a custom resource whose operator was deleted first. kubectl get namespace shop-old -o yaml shows conditions listing what is blocking it.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet