Skip to content

Interview PrepKubernetes

Configuration, Secrets and Storage ​

377. What is a ConfigMap, and how can a Pod consume its data? ​

The short answer

A ConfigMap stores non-secret configuration as keys and values, such as a log level, a feature flag or a whole configuration file, separately from the image. A pod can use it in three ways. It can read it as environment variables, one key at a time or all keys at once. It can mount it as a volume, where each key becomes a file. Or it can build command-line arguments from those variables. The same image then runs with different settings in each environment.

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: checkout-api-config
  namespace: shop-prod
data:
  LOG_LEVEL: info                          # (1)
  PAYMENT_TIMEOUT_MS: "3000"
  rules.json: |                            # (2)
    { "maxItemsPerOrder": 50, "currencies": ["EUR", "GBP"] }
  1. Simple values. All values are text, so numbers are quoted.
  2. A whole file stored under one key.

Three ways to use it:

yaml
containers:
  - name: api
    image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...
    envFrom:
      - configMapRef: { name: checkout-api-config }          # (1)
    env:
      - name: TIMEOUT_MS
        valueFrom:
          configMapKeyRef: { name: checkout-api-config, key: PAYMENT_TIMEOUT_MS }   # (2)
    args: ["--timeout=$(TIMEOUT_MS)"]                        # (3)
    volumeMounts:
      - { name: config, mountPath: /etc/checkout, readOnly: true }
volumes:
  - name: config
    configMap:
      name: checkout-api-config
      items: [{ key: rules.json, path: rules.json }]         # (4)
  1. Every key becomes an environment variable. Keys that are not valid variable names, such as rules.json, are skipped.
  2. One key, under a different variable name.
  3. Kubernetes replaces $(TIMEOUT_MS) in arguments with the variable's value.
  4. Mounted as a file: the app reads /etc/checkout/rules.json.

Limits and options:

  • A ConfigMap holds at most 1 MiB of data. It is for configuration, not for data files.
  • immutable: true forbids changes. That prevents accidents, and lets the kubelet stop watching it, which reduces load in large clusters. Changing configuration then means creating a new ConfigMap with a new name (question 380).
  • Never put secrets in ConfigMaps. Use Secrets, which have separate access control and protection (question 378).

378. What is a Kubernetes Secret, and why is base64 encoding not the same as encryption? ​

The short answer

A Secret is an object for sensitive values, such as passwords, tokens and keys, consumed by pods like a ConfigMap, as variables or files. Its values are stored base64-encoded, which only changes how the bytes are written so any binary value fits in text. Anyone can reverse it in one command, with no key. The real protections around a Secret are access control, encryption at rest in etcd, and keeping it out of places it does not belong.

Base64 is not protection:

bash
kubectl get secret checkout-db -n shop-prod -o jsonpath='{.data.password}'
czNjcjN0LVBhc3N3MHJk
bash
echo czNjcjN0LVBhc3N3MHJk | base64 -d
s3cr3t-Passw0rd

Encoding turns data into another form so it can be stored or sent safely, and anyone can turn it back. Encryption turns data into a form that can only be turned back with a secret key. Base64 is encoding. It exists so that binary values, such as a certificate's key file, can sit inside YAML and JSON.

How Secrets differ from ConfigMaps, in practice:

ConfigMapSecret
Intended forSettingsCredentials and keys
Values stored asPlain textbase64 text
Access controlUsually readable by manyShould be readable by few
Encryption at rest in etcdRarelyShould be turned on
On the node, when mountedWritten to diskKept in memory, a tmpfs

Types you will see:

TypeHolds
OpaqueAny keys and values, the default
kubernetes.io/tlsA TLS certificate and key
kubernetes.io/dockerconfigjsonRegistry pull credentials (question 354)
kubernetes.io/service-account-tokenA long-lived service account token, now discouraged

What actually keeps a Secret safe is the subject of question 379: access control, encryption at rest, and ideally keeping the real value in an outside secret manager.

379. How should sensitive configuration be protected with RBAC, encryption at rest, and an external secret manager? ​

The short answer

Use all three. Access control decides who may read Secrets: only the workloads and the few people who need them, remembering that permission to list Secrets reveals all their values. Encryption at rest makes sure Secrets stored in etcd and its backups are encrypted, ideally with a key held in a cloud key service. An outside secret manager, such as Azure Key Vault, AWS Secrets Manager or HashiCorp Vault, holds the real values, with auditing and rotation, and a tool brings them into the cluster only where they are needed.

1. Access control (RBAC).

  • Grant get on specific Secrets only to the service accounts that need them (question 416).
  • Be careful with list and watch: they return the full contents of every Secret in the namespace, not just names.
  • Avoid wildcard roles, such as resources: ["*"], which include Secrets without anyone noticing.
  • Remember the other paths to a Secret. Anyone who can create a pod in a namespace can mount any Secret in that namespace into it. Anyone who can exec into a pod can read its mounted secrets. Restrict those permissions too.

2. Encryption at rest. By default, etcd stores Secrets only base64-encoded. The API server can encrypt them before storing:

yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - kms:
          apiVersion: v2
          name: azure-kms
          endpoint: unix:///opt/azurekms.socket      # (1)
      - identity: {}                                 # (2)
  1. KMS encryption: each Secret is encrypted with a data key, and that data key is itself encrypted by a master key that never leaves the cloud's key service. This is called envelope encryption (question 551).
  2. The fallback, identity, means unencrypted, used only to read data written before encryption was turned on.

On managed Kubernetes you switch this on with a cluster setting rather than writing the file, for example AKS's KMS etcd encryption with Azure Key Vault, and EKS's envelope encryption with AWS KMS.

3. An outside secret manager. The manager is the single source of truth: it audits every read, supports rotation and versioning, and keeps values out of Git and out of etcd backups. A tool brings values into pods only where needed (question 492). Pods authenticate to it with workload identity, not a stored password (question 615).

Also: mount secrets as files rather than environment variables where you can (question 381), turn on audit logging of Secret reads, and rotate values on a schedule.

380. What happens when a ConfigMap or Secret changes after it has been mounted or injected into a running Pod? ​

The short answer

Environment variables never change: they were copied into the process when the container started, so the pod must be restarted to see a new value. Files from a mounted volume are updated by the kubelet after a delay, often up to a minute or two, but only if the app re-reads them, and never for files mounted with subPath. Because of this, the usual practice is to roll out new pods whenever configuration changes, by putting a hash of the configuration into the pod template.

How it is usedDoes a running pod see the change?
Environment variableNo, never; restart the pod
Mounted volumeYes, after a delay, if the app re-reads the file
Mounted with subPathNo, never
Immutable ConfigMap or SecretCannot change; create a new one

How file updates work. The kubelet periodically checks mounted ConfigMaps and Secrets. When one changes, it writes the new files into a fresh hidden folder and switches a symbolic link to point at it, so the app never sees a half-written file. The delay depends on the kubelet's sync interval and caches, typically under a couple of minutes.

Most apps only read configuration at start-up, so even updated files change nothing until the app reloads. Some can watch their files, or reload on a signal such as SIGHUP, as nginx does.

The reliable pattern: change configuration, roll out pods.

  1. A checksum in the pod template. Helm charts commonly add an annotation such as checksum/config containing a hash of the ConfigMap's content. When the content changes, the hash changes, so the pod template changes, which starts a normal rolling update.
  2. A new name per version. Kustomize's configMapGenerator creates names such as checkout-api-config-7k2m9h, with a hash suffix, and updates the Deployment to use the new name. Old pods keep the old ConfigMap until they are replaced, which also makes rollbacks clean.
  3. A reloader controller, a small tool that watches ConfigMaps and Secrets and restarts the Deployments that use them.

Why rolling out is better than live reloading: a bad configuration value then goes through the same gradual, health-checked rollout as a bad image, instead of hitting every pod at once (question 422).

381. What is the difference between environment-variable configuration and mounted configuration files? ​

The short answer

Environment variables are simple, read once at start-up, and good for small, flat settings, but they never update, are easy to leak, and cannot hold structured data well. Mounted files can hold whole structured configuration files and certificates, can be updated without restarting the pod, can have strict file permissions, and keep secrets out of places where environment variables tend to leak. Use variables for simple settings, and files for secrets, certificates and structured configuration.

Environment variablesMounted files
Good forSmall, flat valuesStructured files, certificates, secrets
ReadOnce, at start-upWhenever the app reads the file
Updated in a running podNeverYes, after a delay
Visible through/proc/<pid>/environ, crash reports, child processes, debug outputOnly processes that can read the file
PermissionsNoneFile mode, such as 0400
SizePractical limitsLarger, up to the object's 1 MiB

Why secrets are safer as files. Environment variables travel further than people expect:

  1. Child processes inherit them. Every program the app starts gets every secret.
  2. Crash and debug tools print them. Many error reporters attach the whole environment to each report.
  3. They appear in kubectl describe and container inspection when set directly as values.

A secret in a file, mounted read-only with mode 0400, is only read when the app opens that file.

yaml
volumeMounts:
  - { name: db-creds, mountPath: /run/secrets/db, readOnly: true }
volumes:
  - name: db-creds
    secret:
      secretName: checkout-db
      defaultMode: 0400

The app reads /run/secrets/db/password. Many libraries and official images support a _FILE convention, such as POSTGRES_PASSWORD_FILE, for exactly this reason (question 128).

382. What is a volume in Kubernetes, and what problem does it solve beyond the container filesystem? ​

The short answer

A container's own filesystem is temporary: when the container restarts, even inside the same pod, everything it wrote is lost. A volume is a folder mounted into a pod whose life is separate from any single container. Some volume types last as long as the pod, which lets containers share files and survive restarts. Others, persistent volumes, outlive the pod entirely. Volumes also deliver configuration, secrets and pod information as files.

The problem. A container that crashes and is restarted by the kubelet gets a fresh writable layer. Anything it wrote, such as a cache or a half-processed upload, is gone. And two containers in one pod cannot see each other's files at all, except through a volume.

Common volume types:

TypeLives as long asUse for
emptyDirThe podScratch space, sharing files between containers
emptyDir with medium: MemoryThe pod, in RAMFast temporary files, counted against memory
configMap, secretThe pod, refreshed from the objectConfiguration and credentials as files
projectedThe podSeveral sources in one folder, including service account tokens
downwardAPIThe podThe pod's own name, labels or limits as files
persistentVolumeClaimBeyond the podDatabases and any data that must survive (question 383)
hostPathThe nodeNode agents only; dangerous for apps
CSI, inlineDepends on the driverSpecial storage, such as secrets from a vault (question 492)

hostPath deserves a warning. It mounts a folder from the node itself. An app using it depends on one particular node, and if it can write to sensitive host paths, it can take over the node. Pod security policies forbid it for ordinary workloads (question 499).

383. What are PersistentVolumes, PersistentVolumeClaims, and StorageClasses? ​

The short answer

A PersistentVolume is a real piece of storage made available to the cluster, such as a cloud disk. A PersistentVolumeClaim is a request for storage made by an application: "I need 100 GiB, writable by one node, of this kind". A StorageClass describes a kind of storage, such as fast SSD disks, and lets claims be fulfilled automatically by creating new volumes on demand. The claim binds to one volume, and pods use the claim, never the volume directly.

namespace shop-prodpodorders-db-0PersistentVolumeClaim100Gi, ReadWriteOnce, premium-ssdcluster-wideStorageClass premium-ssddriver, disk type, reclaim policyPersistentVolumea real 100Gi cloud disknamescreatesbinds
Order, catalogue and delivery. The claim is the order, the StorageClass is the catalogue entry it orders from, and the PersistentVolume is the disk that gets delivered. The pod only ever names its claim.
yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: premium-ssd
provisioner: disk.csi.azure.com               # (1)
parameters: { skuName: Premium_LRS }          # (2)
reclaimPolicy: Retain                         # (3)
volumeBindingMode: WaitForFirstConsumer       # (4)
allowVolumeExpansion: true                    # (5)
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: orders-data
  namespace: shop-prod
spec:
  storageClassName: premium-ssd
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 100Gi } }
  1. The CSI driver that creates the storage. CSI, the Container Storage Interface, is the standard plug-in system that lets any storage vendor work with Kubernetes. This one creates Azure managed disks.
  2. Driver-specific options, here premium SSD disks.
  3. What happens to the disk when the claim is deleted (question 388).
  4. Create the disk only when a pod using the claim is scheduled, in that pod's zone (question 384).
  5. Allow growing the claim later by editing its requested size.

Why the split into three objects? It separates jobs. App teams write claims, saying what they need. Platform teams write StorageClasses, saying what kinds exist and how they are made. Volumes are the plumbing in between, usually created automatically.

384. How does dynamic volume provisioning work? ​

The short answer

When a claim names a StorageClass and no suitable volume exists, the StorageClass's CSI driver creates new storage on demand. Its controller sees the unbound claim, calls the cloud's API to create a disk of the requested size and type, and creates a PersistentVolume for that disk, which binds to the claim. When a pod using the claim starts on a node, the driver attaches the disk to that node and mounts it into the pod. With WaitForFirstConsumer, the disk is created only after the pod is scheduled, so it lands in the pod's zone.

Step by step:

  1. A claim is created, asking for 100 GiB of premium-ssd.
  2. With WaitForFirstConsumer, the claim waits as Pending until a pod that uses it is scheduled. The scheduler picks a node, say in zone 2.
  3. The driver's provisioner, a controller watching claims, calls the cloud to create a 100 GiB premium disk in zone 2.
  4. It creates a PersistentVolume describing that disk, and binds it to the claim. The claim becomes Bound.
  5. When the pod starts, the driver's controller asks the cloud to attach the disk to the pod's node.
  6. The driver's node plug-in, a DaemonSet on each node, formats the disk the first time and mounts it into the pod's folder.

Why WaitForFirstConsumer matters. Cloud disks belong to one availability zone and can only attach to machines in that zone. With the other mode, Immediate, the disk might be created in zone 1 before the scheduler has chosen a node. If the pod then needs to run in zone 2, because zone 1 is full or the pod's rules say so, it can never start. Waiting lets the scheduling decision come first.

What can go wrong, and the claim's events say which:

  • The cloud quota for disks is exhausted.
  • The StorageClass or driver name is wrong, so nothing ever handles the claim.
  • No default StorageClass exists and the claim named none.
  • The disk is still attached to an old node during a move, which can delay a pod's start by a minute or more.

385. What is the difference between ReadWriteOnce, ReadOnlyMany, and ReadWriteMany access modes? ​

The short answer

They say how a volume may be mounted. ReadWriteOnce means read-write by one node at a time, which suits block disks used by a single database. ReadOnlyMany means read-only by many nodes. ReadWriteMany means read-write by many nodes at once, which needs a shared network filesystem, such as NFS, Azure Files or Amazon EFS. ReadWriteOncePod limits a volume to a single pod. Which modes are possible depends entirely on the kind of storage.

ModeShortMounted byTypical storage
ReadWriteOnceRWOOne node, read-writeCloud block disks: Azure Disk, Amazon EBS
ReadOnlyManyROXMany nodes, read-onlyShared filesystems, some snapshots
ReadWriteManyRWXMany nodes, read-writeNFS, Azure Files, Amazon EFS, CephFS
ReadWriteOncePodRWOPOne pod, read-writeDrivers that support it

The detail most people get wrong about RWO. It means one node, not one pod. Two pods on the same node can both mount an RWO volume. If you need to guarantee that only one pod ever writes, for example for a database's files, use ReadWriteOncePod, which became stable in Kubernetes 1.29.

Why block disks are RWO. A block disk is like a hard drive: the filesystem on it assumes one machine controls it. Two machines writing to the same ordinary filesystem at once would corrupt it.

RWX has costs. Network filesystems are slower than local or block storage, especially for many small files and for database workloads. Use them for shared files, such as uploaded images served by many web pods, not for database data.

Many apps do not need RWX at all. Uploaded files often belong in object storage, such as Azure Blob storage or Amazon S3, accessed through an API (question 571).

386. How do StatefulSets use persistent volume claims to preserve stable storage identity? ​

The short answer

A StatefulSet has a claim template. For each pod, it creates a claim with a predictable name made from the template name, the StatefulSet name and the pod's number, such as data-orders-db-1. Pod orders-db-1 always uses that same claim, so if the pod is deleted or moved to another node, the new orders-db-1 reattaches to the same volume and finds its own data. The claims are kept by default even when the StatefulSet is scaled down or deleted.

yaml
spec:
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        storageClassName: premium-ssd
        accessModes: [ReadWriteOnce]
        resources: { requests: { storage: 100Gi } }
bash
kubectl get pvc -n shop-prod
NAME               STATUS   VOLUME          CAPACITY   ACCESS MODES   STORAGECLASS
data-orders-db-0   Bound    pvc-2d1a...     100Gi      RWO            premium-ssd
data-orders-db-1   Bound    pvc-8b7e...     100Gi      RWO            premium-ssd
data-orders-db-2   Bound    pvc-c4f0...     100Gi      RWO            premium-ssd

What happens when orders-db-1's node fails:

  1. The pod is gone. The StatefulSet creates a new pod with the same name, orders-db-1.
  2. The new pod uses the claim data-orders-db-1, by name.
  3. The claim's disk is detached from the failed node and attached to the new node.
  4. The database starts with its own data, as if it had simply restarted.

One constraint: a cloud block disk lives in one zone, so the new pod must be scheduled in that zone. If that zone has no capacity, the pod waits.

Scaling down and deleting. By default, scaling from three to two removes orders-db-2, but keeps data-orders-db-2, so scaling back up gets the same data back. Deleting the StatefulSet also keeps the claims. The persistentVolumeClaimRetentionPolicy field can change this, deleting claims when the StatefulSet is deleted or scaled down, if you prefer automatic clean-up over safety.

387. What happens to a PVC and its underlying storage when a workload is deleted? ​

The short answer

Usually nothing, at first. A claim is its own object, so deleting a Deployment or a StatefulSet does not delete its claims, and the volumes and disks remain. Only when the claim itself is deleted does the volume's reclaim policy decide the disk's fate: Delete removes the cloud disk and its data, Retain keeps them. Deleting a namespace deletes every claim in it, so with the common Delete policy, deleting a namespace can destroy its data.

You deleteClaimsVolumes and disks
A Deployment using a claimKeptKept
A StatefulSetKept, by defaultKept
A claimDeletedFollow the reclaim policy
A namespaceDeleted, all of themFollow the reclaim policy

The reclaim policy at the moment the claim goes:

  • Delete: the volume object and the actual cloud disk are deleted. The data is gone. This is the default for most dynamically provisioned storage.
  • Retain: the volume becomes Released. The disk and its data stay, and someone must decide later what to do with them (question 388).

A protection that helps. A claim still used by a running pod cannot be deleted immediately. It waits in Terminating, held by the kubernetes.io/pvc-protection finalizer, until no pod uses it (question 487). That stops a careless delete from pulling a disk out from under a running database. But once the pod stops, the deletion goes ahead.

Rules that prevent data loss:

  1. Use Retain for important data, such as databases.
  2. Take backups or volume snapshots, separate from the volume itself.
  3. Be very careful with kubectl delete namespace in anything holding data.

388. What are storage reclaim policies, and how can they affect data retention? ​

The short answer

A reclaim policy says what happens to a persistent volume, and its real storage, when the claim using it is deleted. Delete removes the storage, which is convenient for temporary data but loses it permanently. Retain keeps the volume and its data, marked Released, for a person to recover, back up or clean up. The policy usually comes from the StorageClass, where the default is Delete, so important data needs a Retain class, or a policy change on its volume, and backups either way.

PolicyAfter the claim is deletedUse for
DeleteVolume and real storage removedTemporary or easily rebuilt data
RetainVolume kept as Released, data intactDatabases and anything valuable
RecycleA simple wipe, then reuseDeprecated; do not use

Changing the policy on an existing volume:

bash
kubectl patch pv pvc-2d1a... -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

This is a good first step before any risky operation, such as migrating a database to a new StatefulSet.

Recovering a Released volume. A retained volume still remembers its old claim, so a new claim cannot bind to it straight away. To reuse it:

  1. Back up the data first, if in doubt.
  2. Remove the old claim reference from the volume: clear its spec.claimRef.
  3. Create a new claim that asks for that volume by name, with spec.volumeName.

Reclaim policies are not backups. Retain protects against deleting a claim. It does not protect against a bug that corrupts the data, a deleted cloud disk, or a regional outage. For those you need volume snapshots, point-in-time copies of the disk taken through the CSI driver, and real backups stored elsewhere (question 502).

492. What are Sealed Secrets, the External Secrets Operator, and the Secrets Store CSI Driver, and how do they differ? ​

The short answer

All three keep real secret values out of Git and out of hand-made Kubernetes Secrets, in different ways. Sealed Secrets encrypts a secret with a key that only the cluster holds, so the encrypted version can safely live in Git, and a controller in the cluster turns it back into a normal Secret. The External Secrets Operator keeps the real value in an outside manager, such as Key Vault, and copies it into a Kubernetes Secret, refreshing it regularly. The Secrets Store CSI Driver mounts values from the outside manager straight into pods as files.

Sealed SecretsSealedSecret in Git, encryptedcontroller decryptsKubernetes SecretExternal Secrets OperatorKey Vault holds valueoperator copies itevery refresh intervalKubernetes SecretSecrets Store CSI DriverKey Vault holds valuedriver fetches itwhen the pod startsfile inside the podOnly the first stores anything secret in Git, and only in encrypted form.
Three routes for a secret. Sealed Secrets makes Git safe for encrypted secrets. The other two keep the value in an outside manager and only bring it into the cluster when and where it is needed.

Sealed Secrets:

bash
kubectl create secret generic checkout-db --from-literal=password='s3cr3t' --dry-run=client -o yaml \
  | kubeseal --format yaml > checkout-db.sealed.yaml

kubeseal encrypts the Secret with the cluster controller's public key. Only the controller's private key, inside the cluster, can decrypt it. The output is safe to commit. The catch: back up the controller's private key. Lose it, and every sealed secret in Git is unreadable.

The External Secrets Operator:

yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: checkout-db
  namespace: shop-prod
spec:
  refreshInterval: 1h                               # (1)
  secretStoreRef: { kind: ClusterSecretStore, name: azure-key-vault }
  target: { name: checkout-db }                     # (2)
  data:
    - secretKey: password
      remoteRef: { key: checkout-db-password }      # (3)
  1. Re-read the value from the vault every hour, so a rotated password reaches the cluster.
  2. The Kubernetes Secret it creates and maintains.
  3. The secret's name in Key Vault.

The Secrets Store CSI Driver: a SecretProviderClass lists which vault secrets to fetch. The pod mounts a CSI volume using it, and the driver fetches the values, using the pod's workload identity, when the pod starts, writing them as files in memory. It can optionally also create a Kubernetes Secret, for apps that need environment variables.

Sealed SecretsExternal Secrets OperatorSecrets Store CSI Driver
Source of truthEncrypted file in GitOutside secret managerOutside secret manager
Creates a Kubernetes SecretYesYesOptional
Picks up rotationWhen Git changesOn its refresh intervalWhen pods restart, or periodic rotation if enabled
Needs an outside managerNoYesYes
Main riskLosing the controller's keySecret copied into etcdApp must read files

Another option you may hear: SOPS, a tool that encrypts the values inside YAML files with a cloud key or a personal key, often used with GitOps tools that can decrypt during deployment (question 511).

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet