Appearance
Registries and Image Security
130. How do you tag and push an image to a private registry?
The short answer
Log in to the registry, give the image a name that starts with the registry's address, and push it. For example: docker login myacr.azurecr.io, then docker tag checkout-api:1.4 myacr.azurecr.io/shop/checkout-api:1.4, then docker push myacr.azurecr.io/shop/checkout-api:1.4. In CI, log in with a short-lived identity instead of a stored password, and build and push in one step.
bash
az acr login --name myacr # (1)
docker tag checkout-api:1.4 myacr.azurecr.io/shop/checkout-api:1.4 # (2)
docker tag checkout-api:1.4 myacr.azurecr.io/shop/checkout-api:3f9c2ab
docker push myacr.azurecr.io/shop/checkout-api:1.4 # (3)
docker push myacr.azurecr.io/shop/checkout-api:3f9c2ab- Logs Docker in to an Azure Container Registry using your Azure login, which hands Docker a short-lived token. For other registries,
docker login registry.example.comasks for a user name and password or token. docker tagdoes not copy anything. It adds a second name pointing at the same image. The name's first part,myacr.azurecr.io, tells Docker which registry it belongs to.shop/checkout-apiis the repository. Two tags are added: the release version and the commit ID.- Pushes. Docker uploads only the layers the registry does not already have, then the manifest. The second push uploads nothing but a new tag.
The push output ends with the digest:
1.4: digest: sha256:41c9b0e8... size: 1789Record that digest. It is the exact identity of what you just published (question 35).
In CI, build and push in one step, and log in with the pipeline's own identity rather than a stored password. Question 269 explains how OIDC makes that possible.
bash
docker buildx build -t myacr.azurecr.io/shop/checkout-api:${GIT_SHA} --push .131. What is the difference between a registry, repository, tag, and digest?
The short answer
A registry is the server that stores images, such as myacr.azurecr.io. A repository is one named collection of versions of one image inside it, such as shop/checkout-api. A tag is a human-friendly, movable label for one version, such as 1.4. A digest is the fixed fingerprint of one exact image, such as sha256:41c9..., which can never point at anything else.
The full name of an image puts the pieces together:
myacr.azurecr.io / shop/checkout-api : 1.4 @ sha256:41c9b0e8...
registry repository tag digest| Term | Example | Can change? | Analogy |
|---|---|---|---|
| Registry | myacr.azurecr.io | Not applicable | A library building |
| Repository | shop/checkout-api | Not applicable | One shelf of one book |
| Tag | 1.4 | Yes, can be moved | A sticky label on a copy |
| Digest | sha256:41c9... | Never | The book's exact fingerprint |
Two details interviewers like:
- Some registries add a level above repositories. Docker Hub calls it a namespace: your user or organisation name.
library/is the namespace for official images. - A repository name may contain slashes, such as
shop/payments/adapter. The slashes are only part of the name; there is no real folder structure.
132. How do registry authentication and image-pull authentication differ from application runtime credentials?
The short answer
They belong to different users, for different jobs. Push credentials let the CI pipeline write images into the registry. Pull credentials let the machine that runs containers, such as a Kubernetes node, read images from it. Application runtime credentials, such as a database password, are used by your app while it runs. The app never sees the pull credentials, and the registry never sees the app's credentials. Mixing them up leaks access.
| Push credentials | Pull credentials | App runtime credentials | |
|---|---|---|---|
| Who uses them | The CI pipeline | The container runtime, the kubelet | Your app's code |
| What for | Uploading images | Downloading images | Database, APIs, queues |
| Needed when | Build time | Container start | The whole time it runs |
| Permission level | Write to some repositories | Read only | Whatever the app needs |
| Best form | Short-lived OIDC token | The node's or cluster's cloud identity | Secrets from a vault, or a workload identity |
How pull credentials work in Kubernetes. The kubelet, the agent on each node that starts containers, pulls the image before the container exists. So the credentials must be given to the kubelet, through an imagePullSecrets entry or, better, a cloud identity attached to the cluster, such as AKS's managed identity with the AcrPull role. Your app's process never receives them (question 354).
How app credentials work. They are delivered into the running container, as mounted files or environment variables, from a secret store (question 144).
Common mistakes:
- One powerful account for everything. If the same registry password can both push and pull, and it leaks from a cluster, an attacker can overwrite your images. Give clusters read-only access.
- Registry credentials inside the image. An image is readable by everyone who can pull it (question 80).
- Storing a long-lived push password in CI. Use short-lived tokens (question 268).
133. How would you promote an image through development, testing, staging, and production?
The short answer
Build the image once, push it, and from then on move that same digest forward. Each environment's deployment points at the digest, and promotion means updating the next environment's configuration to the same digest after its checks pass: tests in development and testing, approval and checks for staging and production. You may add tags such as staging or copy the image into a production registry, but you never rebuild it.
A typical flow:
- CI builds once, after merge, and pushes
checkout-api:3f9c2ab, the commit ID. It records the digest. - Development deploys that digest automatically and runs quick checks.
- Testing gets the same digest after the development checks pass, and runs longer integration and end-to-end checks.
- Staging gets it, often after a human or an automated gate, and is the final rehearsal on production-like settings.
- Production gets it after approval and a final check that the image is signed and scanned.
What "promoting" looks like in practice. In a GitOps setup, promotion is a pull request that changes the digest in the next environment's configuration file:
yaml
# deploy/staging/checkout-api.yaml
image: myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...Many teams also add a tag to mark progress, without changing the bytes:
bash
docker buildx imagetools create -t myacr.azurecr.io/shop/checkout-api:1.4.2 \
myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8...This creates a new tag pointing at the existing digest, entirely inside the registry. Nothing is downloaded or rebuilt.
Separate production registry? Some companies keep production images in their own registry that only the release pipeline can write to. Promotion then copies the image across, for example with az acr import or a tool called crane. Copying keeps the digest identical.
134. Why should the same immutable image be promoted across environments instead of rebuilding it separately in each environment?
The short answer
Because a rebuild is not the same thing. Between two builds, a base image can be updated, a dependency can resolve to a new version, a build tool can change, or a mirror can serve different bytes. Then the image in production is one nobody tested. Promoting the same digest guarantees that what passed every check is exactly what customers run. It is also faster, keeps one signature valid all the way, and makes rollback simple.
What can differ between two builds of the same commit:
- The base image.
FROM node:20-slimtoday and next week can be two different images (question 87). - Dependencies. Anything not pinned by a lock file can resolve to a newer version. Even with lock files, system packages from
apt-getusually are not pinned. - The build environment. A newer compiler, a different BuildKit version, a different CPU type on the CI machine.
- The network. A package server can return different bytes, or be compromised during one of the builds.
Each is unlikely on a given day. Over hundreds of builds, they happen. And when they do, you get the worst kind of bug: "staging was fine", because staging ran a different image.
What promotion gives you:
- Confidence. Every test result applies to the exact bytes in production.
- Speed. Promotion is a pointer change, seconds instead of a full build.
- Supply-chain proof. The image is signed and its provenance recorded once, in CI (question 146). Every environment can verify the same signature. A rebuilt image would need new, separate proof.
- Simple rollback. Rolling back is pointing at the previous digest, which is already in the registry and already proven.
What must differ between environments is configuration, such as database addresses, replica counts and feature flags. Keep that outside the image, and inject it at deploy time (question 226).
135. How would you handle image retention, cleanup, replication, and registry availability?
The short answer
Write retention rules: delete untagged leftovers after a few days, keep the last several builds of every branch, and never delete anything that is deployed or might be a rollback target. Replicate the registry to every region where you run workloads, so pulls are fast and survive a regional outage. And treat the registry as critical infrastructure, because without it no new container can start, including the ones that replace failed ones.
1. Retention and cleanup. Registries fill up fast: every commit pushes a new image. A sensible policy:
| Image type | Rule |
|---|---|
| Untagged manifests | Delete after 7 days |
| Branch or pull request builds | Keep the last 10 per branch, delete after 30 days |
| Release versions | Keep for at least the support period |
| Anything running in any environment | Never delete |
Most registries can enforce rules like these themselves. Amazon ECR has lifecycle policies. Azure Container Registry has a retention policy for untagged manifests and scheduled purge tasks. Harbor has tag retention rules.
Before deleting, check what is running. A script that lists the digests used by every cluster, and protects them, prevents the worst cleanup accident: deleting the image a node needs to restart a pod.
2. Replication. If your clusters run in Europe and Asia, but the registry is only in Europe, Asian nodes pull across the world on every scale-up. Geo-replication, such as Azure Container Registry's premium tier or Amazon ECR's cross-region replication, keeps a copy in each region, so pulls are local and survive a regional failure.
3. Caching public images. Pulling straight from Docker Hub in production risks rate limits (question 477) and outages beyond your control. A pull-through cache, such as ECR's pull-through cache or ACR's artifact cache, or simply importing approved base images into your own registry, removes that dependency.
4. Availability. The registry sits in the path of every new container. During an incident, when nodes fail and pods must restart elsewhere, a broken registry turns a small outage into a big one. So:
- Use the registry's zone-redundant option.
- Monitor pull errors and pull latency, not only whether the registry's website is up.
- Remember that nodes keep images they have already pulled, so existing pods keep running during a registry outage, but new ones cannot start (question 446).
136. How do you scan a container image for operating-system and application dependency vulnerabilities?
The short answer
Run an image scanner such as Trivy, Grype, Docker Scout or your registry's built-in scanner. It unpacks the image, reads the list of installed operating system packages and the language packages, such as npm, Python and Java libraries, and compares every name and version against databases of known vulnerabilities. Run it in CI on every build, fail the build on serious fixable findings, and rescan stored images regularly, because new vulnerabilities are found every day.
First, two terms. A CVE, Common Vulnerabilities and Exposures, is a public ID for one known security flaw, such as CVE-2024-3094. Its severity is usually given as a CVSS score from 0 to 10, grouped into low, medium, high and critical.
How a scanner works:
- It unpacks the image's layers and builds the final filesystem.
- It finds the operating system packages by reading the package manager's own records, such as
/var/lib/dpkg/statuson Debian or/lib/apk/db/installedon Alpine. - It finds the language packages by reading files such as
package-lock.json, Python's installed package folders, Java JAR files, and the build information that Go writes into its binaries. - It matches every package and version against vulnerability databases, such as the distributions' own security trackers and the GitHub Advisory Database.
- It reports each match: the CVE, the severity, the installed version, and the version that fixes it, if any.
bash
trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 \
myacr.azurecr.io/shop/checkout-api:3f9c2abcheckout-api:3f9c2ab (debian 12.7)
Total: 1 (HIGH: 1, CRITICAL: 0)
┌─────────┬────────────────┬──────────┬───────────────────┬───────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │
├─────────┼────────────────┼──────────┼───────────────────┼───────────────┤
│ libssl3 │ CVE-2024-XXXXX │ HIGH │ 3.0.14-1~deb12u1 │ 3.0.15-1~deb12u1 │
└─────────┴────────────────┴──────────┴───────────────────┴───────────────┘--severity HIGH,CRITICALreports only serious findings.--ignore-unfixedskips findings that have no fix yet, since there is nothing you can update to. Track those separately.--exit-code 1makes the command fail when it finds something, which fails the pipeline step.
Where to scan:
- In CI, on every image built, so problems are caught before they ship.
- In the registry, continuously. An image that was clean on Monday can have a critical CVE on Thursday, because the flaw was only discovered on Wednesday.
- In running clusters, so you know what is affected in production right now.
The usual fix is not a clever patch. It is rebuilding on an updated base image, or bumping the dependency, then redeploying.
137. How do you decide whether a vulnerability finding should block a release?
The short answer
Use a written policy, not a feeling. Block when a finding is serious, can actually be exploited in your situation, and has a fix available. Consider the severity, whether it is known to be exploited in the wild, whether the vulnerable code is even used by your app, whether the service is exposed to the internet, and whether a fixed version exists. Anything you let through gets a recorded exception with an owner and an expiry date.
A raw scanner report for a large image can list hundreds of findings. Blocking on all of them blocks every release forever. Blocking on none makes the scanner pointless. So decide by risk, using a few questions:
- How bad is it? The severity score, critical or high.
- Is it being exploited? Lists such as CISA's KEV, the Known Exploited Vulnerabilities catalogue, name flaws attackers are actively using. A score called EPSS estimates how likely exploitation is in the next month.
- Can it be reached? Is the vulnerable package actually loaded and used at run time? A flaw in a command-line tool your app never runs is far less urgent than a flaw in your web framework.
- Is the service exposed? An internet-facing API is more at risk than a batch job on a private network.
- Is there a fix? If no fixed version exists, blocking does not make you safer. You need another control instead.
A policy that works for many teams:
| Finding | In production images |
|---|---|
| Critical, fix available | Block |
| Known to be exploited, any severity | Block |
| High, fix available, internet-facing | Block |
| High, fix available, internal | Fix within 14 days |
| No fix available | Record, watch, add other protection |
| Low or medium | Fix in normal updates |
Recording decisions. When a finding is accepted, record it with a reason, an owner and an expiry date (question 279). For findings that do not affect you, a VEX document, Vulnerability Exploitability eXchange, is a machine-readable statement such as "CVE-X does not affect this image, because the vulnerable code is never used". Scanners can read it and stop reporting that finding.
Why they ask: they want to hear judgment. "We block everything" and "we look at it later" are both wrong answers.
138. What is the principle of least privilege for containers?
The short answer
Give each container only the permissions it needs to do its job, and nothing more, so that if it is broken into, the attacker can do as little as possible. In practice, run as a non-root user. Drop all Linux capabilities, and add back only the ones you need. Use a read-only filesystem. Never use privileged mode or mount the Docker socket. Limit the container's network access and resources, and give it only the cloud and cluster permissions its own job requires.
The idea is old and simple: assume any part of your system might one day be broken into, and limit how much damage that part could do. A container is just a process, so it should have no more power than the job requires.
A checklist, from the inside out:
- User. Run as a non-root user with a fixed number (question 139).
- Capabilities.
--cap-drop ALL, then add back only what is needed, often nothing (question 101). - Privilege escalation.
--security-opt no-new-privilegesstops a process gaining more power through specialsetuidprograms. - Filesystem.
--read-only, with small writable mounts only where needed (question 96). - System call filter. Keep Docker's default seccomp profile on. Never disable it.
- No dangerous mounts. No Docker socket, no host root folder, no
/procor/sysfrom the host (question 140). - No privileged mode (question 141).
- Network. Only the networks it needs, and no published ports unless required (question 121).
- Resources. Memory, CPU and process limits, so one container cannot starve the host.
- Image. As little inside as possible: no shell, no package manager, no compilers (question 143).
- Identity. Its cloud and Kubernetes permissions cover only its own resources, such as reading one secret or one storage bucket (question 416).
bash
docker run -d --user 10001:10001 --cap-drop ALL --security-opt no-new-privileges \
--read-only --tmpfs /tmp:size=32m --memory 512m --pids-limit 200 \
--network back shop/api:1.4139. Why should an application container run as a non-root user where possible?
The short answer
Because, without user namespaces, root inside a container is the same root user as on the host, held back only by capabilities and filters. If an attacker breaks into a root container, every container-escape bug and every risky mount becomes a path to owning the whole host. A non-root user cannot do most of that: it cannot change system files, install software, or use most escape techniques. It costs almost nothing to do.
What root in a container can do that a normal user cannot:
- Exploit escape bugs. Many real container-escape flaws, such as the 2019 runc flaw CVE-2019-5736, worked only when the process inside was root.
- Abuse mounts. If a host folder is mounted in, root can change any file in it, including files the host trusts, such as startup scripts.
- Change the container. Install tools with the package manager, replace the app's own files, or start listening on any port.
- Use any capability left on. The capabilities Docker keeps by default apply to root, and are what make some attacks possible.
How to run as non-root:
dockerfile
RUN groupadd --system --gid 10001 app \
&& useradd --system --uid 10001 --gid app --no-create-home app
COPY --chown=10001:10001 dist/ /app/dist/
USER 10001:10001
EXPOSE 8080- Create the user with a fixed number, so permissions and Kubernetes policies can rely on it (question 57).
- Make files the app must write belong to that user, using
COPY --chown. - Listen on a port above 1024. Ports below 1024 traditionally need root. Use 8080 inside the container, and map port 80 or 443 outside.
Many official images already include a non-root user, such as node in the Node.js images and nonroot in distroless images.
In Kubernetes, runAsNonRoot: true makes the kubelet refuse to start a container whose user is root, and the "restricted" Pod Security level requires it (question 499).
140. What are the risks of mounting the Docker socket into a container?
The short answer
Whoever can talk to the Docker socket controls the Docker daemon, and the daemon runs as root. So a container with the socket mounted can start a new privileged container with the host's whole disk mounted in, and become root on the host in one command. It can also read every other container's environment and secrets, stop them, and replace images. Mounting the socket is giving away the machine.
People mount the socket for convenient reasons: a CI job that needs to build images, a monitoring tool that lists containers, a dashboard.
bash
docker run -v /var/run/docker.sock:/var/run/docker.sock some/toolNow look at what anything inside that container can do, if it is compromised:
bash
docker run --rm -it --privileged --pid=host -v /:/host alpine chroot /host sh # (1)
docker inspect $(docker ps -q) --format '{{.Name}} {{json .Config.Env}}' # (2)- Starts a new container that is privileged, shares the host's process namespace, and mounts the host's root filesystem at
/host, then switches into it. The result is a root shell on the host itself. - Prints every container's environment variables, including any passwords and keys passed that way.
The attack path in real life: an attacker finds a vulnerability in a web dashboard, or sends malicious code in a pull request run by a CI job that has the socket. Either way, they reach the socket. From there they control the whole host, and every secret on it.
Safer alternatives:
- For building images: rootless BuildKit, Buildah, or a dedicated build machine that runs nothing else (question 469).
- For tools that only need to read, such as monitoring: a socket proxy, a small service that sits in front of the socket and allows only safe, read-only API calls, such as listing containers.
- For deployment tools: use the orchestrator's API with a limited identity instead of the raw Docker API.
141. Why can running a container in privileged mode be dangerous?
The short answer
--privileged switches off almost every protection that makes a container a container. It grants every Linux capability, disables the seccomp filter and the AppArmor or SELinux profile, and gives access to all of the host's devices. A process in a privileged container can mount the host's disks, change kernel settings and load kernel modules, so escaping to the host is trivial. It should only be used for rare system-level tools, and almost always there is a narrower option.
What --privileged changes:
| Protection | Normal container | Privileged container |
|---|---|---|
| Capabilities | About 14 basic ones | All of them |
| Seccomp filter | Default profile | Off |
| AppArmor or SELinux | Default profile | Off |
Host devices in /dev | A few safe ones | All, including disks |
Kernel settings in /sys | Read-only | Writable |
How an escape looks:
bash
docker run --rm -it --privileged alpine sh
# inside the container:
mkdir /host && mount /dev/sda1 /host # mount the host's disk
chroot /host # now at the host's rootFrom there, an attacker can add an SSH key for root, change system files, or install anything. No bug was needed. Privileged mode simply allows it.
Why people use it, and what to use instead:
| Reason given | Narrower alternative |
|---|---|
| "Docker-in-Docker needs it" | Rootless BuildKit or Buildah (question 469) |
| "The app needs one device" | --device /dev/ttyUSB0 |
| "The app needs to manage the network" | --cap-add NET_ADMIN |
| "Something fails without it" | Find the exact capability or system call that is blocked |
In Kubernetes, the equivalent is securityContext.privileged: true, and both the baseline and restricted Pod Security levels forbid it (question 499).
142. What are rootless Docker and user-namespace remapping, and what risks do they reduce?
The short answer
Both use the kernel's user namespaces so that "root" inside a container is not root on the host. With user-namespace remapping, the Docker daemon still runs as root, but each container's user numbers are shifted into a high, unprivileged range on the host. With rootless Docker, the daemon itself runs as an ordinary user, so nothing in the whole stack is root. Either way, an attacker who escapes a container lands as a powerless user on the host.
The mechanism: user namespaces. A user namespace lets a process have one set of user numbers inside and a different set outside. A mapping says, for example: "UID 0 to 65535 inside become UID 100000 to 165535 outside". The process sees itself as root. The kernel treats it, for anything outside its namespace, as user 100000, a nobody.
User-namespace remapping, userns-remap in daemon.json:
json
{ "userns-remap": "default" }Docker creates a user called dockremap and maps every container into its range of user numbers, taken from /etc/subuid and /etc/subgid. The daemon is still root, but container processes are not.
Rootless Docker:
bash
dockerd-rootless-setuptool.sh install
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sockThe daemon, containerd and every container run as your own user account. The Docker socket is in your user's folder, not /var/run.
What risks they reduce:
- Container escapes. An escape now yields an unprivileged user, not root.
- Daemon compromise, for rootless: a flaw in the daemon itself does not give root.
- Risky mounts. A root process in the container can no longer modify host files owned by real root.
The costs:
- File ownership on bind mounts looks strange, because host files belong to unmapped users (question 94).
- Rootless mode has limits: binding ports below 1024 needs extra set-up, networking is slower because it goes through a user-space network stack, and resource limits need cgroup v2 with delegation enabled.
- Some features, such as privileged containers, do not work, which is partly the point.
Kubernetes can also run pods in their own user namespace, with hostUsers: false in the pod specification, which gives the same protection to cluster workloads.
143. How would you reduce the attack surface of a production container image?
The short answer
Ship as little as possible, and make what ships hard to abuse. Use a multi-stage build and a minimal base such as distroless, so there is no shell, package manager or compiler. Include only production dependencies, pinned and patched. Run as a non-root user, keep no secrets in the image, start the container with a read-only filesystem and no capabilities, scan every build, rebuild regularly, and sign what you ship.
Attack surface means everything an attacker could use or attack. Every package, tool, open port and permission is part of it. Reducing it means removing things.
Inside the image:
- Minimal base. Distroless or
scratchinstead of a full distribution (question 70). No shell means an attacker who gets code execution cannot simply open a shell and look around. - Multi-stage build, so compilers and build tools never ship (question 65).
- Production dependencies only (question 67).
- No secrets, ever (question 80).
- No special-permission programs. Programs marked
setuidrun with their owner's power, often root. Remove any you do not need. - Pinned versions and a regularly updated base (question 86).
When it runs:
- Non-root user (question 139).
- Read-only root filesystem (question 96).
- All capabilities dropped,
no-new-privilegesset (question 138). - Only the needed port exposed, and only to the networks that need it.
Around it:
- Scan every build, and rescan stored images (question 136).
- Rebuild often, for example weekly, even without code changes, so base image fixes arrive.
- Sign images and verify signatures before they run (question 147).
A typical result: a 1 GB image with 400 scanner findings becomes a 60 MB image with fewer than ten, none of them in packages the app actually uses.
144. What is the difference between a secret needed during image build and a secret needed at application runtime?
The short answer
A build-time secret is needed only while the image is being built, for example a token to download private packages, and must disappear when the build ends. It is passed with a BuildKit secret mount. A runtime secret is needed by the running app, for example a database password, and is supplied when the container starts, by the platform, from a secret store. Neither should ever be inside the image.
| Build-time secret | Runtime secret | |
|---|---|---|
| Example | Private npm token, Git SSH key | Database password, payment API key |
| Used by | One build step | The running app |
| Lives for | Seconds, one step | The container's whole life |
| How to pass it | RUN --mount=type=secret | Mounted file or variable at start-up |
| Comes from | CI's secret store | A vault, such as Azure Key Vault |
| In the image? | Never | Never |
Build-time is covered in question 79: the secret is mounted into one RUN step and leaves no trace.
Runtime secrets come from outside, every time a container starts:
- Docker and Compose: the
secretsfeature, mounted as files in/run/secrets/(question 128). - Kubernetes: Secret objects mounted as files or variables, or better, synced from a vault by a tool such as the Secrets Store CSI driver (question 492).
- Cloud platforms: the platform reads the vault for the app, using the app's own identity.
Why the split matters. The two kinds have different owners and different lives. A build token belongs to CI, and the production cluster should never see it. A production database password belongs to production, and CI should never see it. Keeping them apart means a leak in one place does not expose the other. And because runtime secrets are injected, not built in, you can rotate a password without rebuilding a single image.
145. How can you prevent private keys or tokens from leaking into image layers, metadata, logs, or build output?
The short answer
Block each route separately. Keep secret files out of the build context with .dockerignore. Pass build secrets only through secret mounts, never through ARG, ENV, COPY or labels. Make sure build commands never print them, and let the CI system mask known secrets in logs. Scan both the source and the finished image for secrets. And if one leaks anyway, rotate it immediately, because deleting the image does not delete the copies already pulled.
Route by route:
| Leak route | How it happens | How to block it |
|---|---|---|
| Layers | COPY .env or a key file, even if deleted later | .dockerignore; secret mounts |
| Image config | ENV API_KEY=... | Never put secrets in ENV |
| History | ARG TOKEN used in a RUN step | Secret mounts instead of ARG |
| Labels | A token put in a LABEL for "metadata" | Labels are public by design |
| Build logs | RUN echo $TOKEN, debug output, set -x | No printing; CI log masking |
| Tool caches | A tool writes the token into its config file inside the layer | Check what each step writes |
Practical habits:
- An allow-list
.dockerignore(question 60), so a new.envor key file is excluded automatically. - No
set -xin build scripts that handle secrets. It makes the shell print every command with its variables filled in. - CI masking. CI systems hide the values of registered secrets in logs, replacing them with
***. They only know about secrets you registered, so a secret built from parts, or encoded differently, slips through. - Scan the finished image for secrets. Trivy's secret scanner looks through every layer for keys and tokens:
bash
trivy image --scanners secret myacr.azurecr.io/shop/checkout-api:3f9c2ab- Scan the source too, before commit and in CI, with a tool such as gitleaks (question 14).
If a secret leaks into an image: treat it as compromised. Rotate it, which means issuing a new one and revoking the old one. Then delete the image and rebuild. Deleting alone is not enough, because every machine and registry that pulled the image still has a copy.
146. What are image signing, SBOMs, and build provenance, and what assurance does each provide?
The short answer
A signature proves who published an image and that it has not changed since. An SBOM, a software bill of materials, is a list of every component and version inside the image, which answers "are we affected?" the day a new vulnerability is announced. Build provenance is a signed record of how the image was made: from which source commit, by which pipeline, on which builder. Together they let you check an image's origin, contents and history before you run it.
1. Signing. A digital signature is made with a private key, and anyone with the matching public key can check it. Signing an image's digest proves two things: the holder of that key signed it, and the image is byte-for-byte what was signed. The most common tool is Cosign, from the Sigstore project. Its keyless mode avoids managing keys at all. The pipeline proves its identity with an OIDC token (question 269), and receives a short-lived certificate for that identity. It signs with it. The signature is recorded in a public log called Rekor, so it can be checked later.
2. SBOM. A software bill of materials lists every package in the image with its version and license, in a standard format such as SPDX or CycloneDX. Its value shows on the day a new vulnerability is announced: instead of rescanning everything, you search your SBOMs and know within minutes which images contain the affected version (question 274).
3. Provenance. A signed statement, usually in the in-toto format and following the SLSA framework, recording: the source repository and commit, the workflow that ran, the builder, and the build's inputs. It answers "did this image really come from our pipeline building our main branch?"
bash
docker buildx build --sbom=true --provenance=mode=max \
-t myacr.azurecr.io/shop/checkout-api:3f9c2ab --push . # (1)
cosign sign myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8... # (2)- BuildKit creates an SBOM and a detailed provenance record during the build and pushes both with the image as attestations, signed statements attached to it.
- Cosign signs the digest. In CI with OIDC available, this is keyless.
| Signature | SBOM | Provenance | |
|---|---|---|---|
| Question answered | Who published it, unchanged? | What is inside? | How, from what? |
| Protects against | Tampered or fake images | Not knowing your exposure | Images built outside your pipeline |
| Used by | Admission checks before deploy | Vulnerability response | Policy checks, audits |
147. How would you verify that a deployed image came from an approved build and has not been replaced?
The short answer
Sign every image in the approved pipeline, deploy by digest, and make the cluster check before running anything. An admission policy verifies that each image's signature was made by your pipeline's identity, for example your repository's release workflow on the main branch, and rejects anything else. Turn on tag immutability in the registry, and regularly compare the digests actually running with your release records.
1. Sign in the approved pipeline only. Using keyless signing, the signature records the pipeline's identity, such as "the workflow release.yml in repository shop/checkout-api, running on branch main". Nobody else can produce a signature with that identity.
2. Verify before running. You can check by hand:
bash
cosign verify myacr.azurecr.io/shop/checkout-api@sha256:41c9b0e8... \
--certificate-identity "https://github.com/shop/checkout-api/.github/workflows/release.yml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com"This succeeds only if a valid signature exists from exactly that workflow, on exactly that branch, issued by GitHub's token service.
3. Make the cluster enforce it. An admission controller is a gatekeeper that Kubernetes calls before accepting any new object (question 486). A policy engine such as Kyverno or the Sigstore policy controller can run the same check on every pod:
yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: only-signed-shop-images
spec:
validationFailureAction: Enforce
rules:
- name: verify-signature
match:
any:
- resources: { kinds: [Pod] }
verifyImages:
- imageReferences: ["myacr.azurecr.io/shop/*"]
attestors:
- entries:
- keyless:
subject: "https://github.com/shop/*/.github/workflows/release.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"A pod using an unsigned image, or one signed by anyone else, is rejected. Even someone with permission to apply YAML by hand cannot run an unapproved image.
4. Remove ways to swap images:
- Deploy by digest, so a moved tag cannot change what runs.
- Turn on tag immutability, so tags cannot be overwritten in the registry.
- Restrict who can push to production repositories.
5. Keep checking. A scheduled job lists the digests running in every cluster and compares them with the release records. Any digest that no pipeline released is an incident (question 454).
148. What is the difference between a vulnerability scanner, a secret scanner, and a static code analyzer?
The short answer
A vulnerability scanner looks for known, published flaws in components you use, such as an old OpenSSL or a vulnerable npm package. A secret scanner looks for credentials that should not be there, such as API keys and private keys, in code, history or images. A static code analyzer reads your own source code and looks for dangerous patterns you wrote, such as SQL built from user input. Each finds things the others cannot.
| Vulnerability scanner | Secret scanner | Static code analyzer | |
|---|---|---|---|
| Looks at | Third-party packages and versions | Text in files, history, layers | Your own source code |
| Finds | Known CVEs | Keys, tokens, passwords | Insecure code patterns |
| How | Matches versions to databases | Patterns and randomness checks | Analyses code structure and data flow |
| Examples | Trivy, Grype, Dependabot | gitleaks, TruffleHog, GitHub secret scanning | Semgrep, CodeQL, SonarQube |
| Typical finding | "lodash 4.17.20 has CVE-2021-23337" | "AWS key in config.js line 12" | "user input reaches a SQL query unescaped" |
Vulnerability scanners work on what you import. They know nothing about bugs in your own code. Their technical names are SCA, software composition analysis, for libraries, and image scanning for whole images (question 214).
Secret scanners look for things that look like credentials: fixed patterns, such as AWS keys starting with AKIA, plus long random-looking strings. They must scan the whole Git history, not only the current files, because a secret committed and then deleted is still in the history.
Static analyzers, also called SAST, static application security testing, follow how data moves through your code. They can spot, for example, that a value from an HTTP request reaches a database query without being made safe, which allows SQL injection.
All three belong in the pipeline, at different points. Secret scanning runs before code is even committed, and on every pull request. Static analysis runs on every pull request. Vulnerability scanning runs on dependencies, on every built image, and continuously in the registry.
477. What are Docker Hub's pull rate limits, and how do you stop them from breaking CI and deployments?
The short answer
Docker Hub limits how many images can be pulled in a time window. According to Docker's documentation in 2025, anonymous pulls are limited to 100 per 6 hours per IP address, and free logged-in users to 200 per 6 hours, while paid accounts are effectively unlimited. CI runners and cluster nodes often share one outgoing IP address, so they hit the limit quickly and fail with toomanyrequests. The fix is to authenticate, and above all to stop pulling from Docker Hub in your critical path by caching or copying images into your own registry.
The error looks like this:
Error response from daemon: toomanyrequests: You have reached your unauthenticated pull rate limit.Why CI and clusters hit it so easily. Limits are counted per outgoing IP address. A company network, a set of hosted CI runners, or a cluster's nodes behind one NAT gateway all share a single address. Fifty builds an hour, each pulling a few base images, use up an anonymous allowance quickly.
Docker has changed these limits several times, so check the current numbers in Docker's documentation before relying on them.
The fixes, best first:
- Do not pull public images in the critical path at all. Copy approved base images into your own registry, or use a pull-through cache. A pull-through cache is a registry feature that fetches an image from Docker Hub the first time, and serves its own copy after that. Amazon ECR and Azure Container Registry both offer one. Builds and nodes then pull only from your registry.
- Authenticate every pull with a paid Docker account, in CI and on nodes.
- Pull less. In Kubernetes,
imagePullPolicy: IfNotPresentwith unique tags avoids pulling an image that is already on the node (question 353). In CI, a persistent builder or registry cache avoids pulling the same base image on every job (question 77).
Why fix 1 is best: it removes a dependency on an outside service from your deployments entirely. If Docker Hub is slow, down, or changes its rules again, your releases and your autoscaling keep working.