Skip to content

Interview PrepDocker

BuildKit, Build Secrets and Reproducible Builds ​

75. How do BuildKit and Buildx improve image building compared with a basic legacy build workflow? ​

The short answer

BuildKit is Docker's modern build engine, and the default since Docker 23. It plans the whole build as a graph, so independent stages run in parallel and unused stages are skipped. It transfers only the files a build needs, adds cache mounts, secret and SSH mounts, cache export to registries, multi-platform builds and build attestations. Buildx is the command-line plugin that drives BuildKit and lets you create and manage builders, including remote ones.

The legacy builder ran a Dockerfile like a script: one line after another, from top to bottom. It uploaded the whole build context first, could not share cache between machines, and had no safe way to pass a secret.

BuildKit first converts the Dockerfile into a graph of steps, where each step knows which earlier steps it needs. That one change unlocks most of its features:

  1. Parallel stages. In a multi-stage build, stages that do not depend on each other build at the same time.
  2. Skipping unused work. If the final stage never copies from a stage, that stage is not built at all.
  3. Smarter file transfer. Only files that steps actually use are sent to the builder, and on later builds only the changes.
  4. Cache mounts. A package manager's download cache can persist between builds without entering any layer (question 76).
  5. Secret and SSH mounts. Credentials reach one step without ever being saved (question 79).
  6. Cache export and import. Cache can be saved to a registry and reused by a brand-new CI machine (question 77).
  7. Multi-platform builds. One command builds for Intel and ARM processors (question 82).
  8. Attestations. It can attach a software bill of materials and a record of how the image was built (question 146).
  9. Build checks. It warns about common mistakes, such as secrets passed through ARG or ENV.

Buildx is how you talk to BuildKit with all of these options. docker build now quietly uses Buildx too, but Buildx adds builders: separate BuildKit instances you can create, choose and share.

bash
docker buildx create --name ci --driver docker-container --use   # (1)
docker buildx ls                                                  # (2)
docker buildx bake                                                # (3)
  1. Creates a builder that runs BuildKit inside its own container. This driver supports every feature, including multi-platform builds and every cache export type.
  2. Lists builders and the platforms each can build for.
  3. bake builds several images at once from one definition file, a little like Docker Compose does for running containers.

76. What are BuildKit cache mounts, and how would you use them for package-manager caches? ​

The short answer

A cache mount gives one RUN step a folder that persists between builds but is never saved into the image. You point it at the package manager's download cache. When the dependency list changes and the layer must be rebuilt, the package manager still finds most packages already downloaded, so the rebuild takes seconds instead of minutes. The image stays small, because the cache is not in any layer.

The layer cache from question 62 is all or nothing. Add one new library to package.json and npm ci runs again from scratch, downloading all 900 packages. A cache mount fixes that.

dockerfile
# syntax=docker/dockerfile:1
FROM node:20-slim
WORKDIR /app
COPY package.json package-lock.json ./
# (1)
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev
  1. --mount=type=cache,target=/root/.npm attaches a persistent folder at /root/.npm, where npm keeps its downloads. The folder is kept by the builder between builds. It is mounted only while this step runs, and its contents are never written into the layer.

Now, when one library is added, the step still reruns, but npm finds 899 packages already in its cache and downloads only one.

The same idea for other package managers:

dockerfile
# Python
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
# Go
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build go build -o /out/app .
# Maven
RUN --mount=type=cache,target=/root/.m2 mvn -B package

For Debian's apt, there is one extra step, because the official images include a setting that deletes downloaded packages after every install:

dockerfile
RUN rm -f /etc/apt/apt.conf.d/docker-clean
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends gcc

sharing=locked means only one build at a time may use this cache. apt breaks if two builds write to its folders at once. The other options are shared, the default, where many builds use it together, and private, where each build gets its own copy.

One limit to remember: cache mounts live inside the builder. On a fresh CI machine they start empty, and the registry cache from question 77 does not include them. Either keep a long-lived builder, or accept that cache mounts mainly speed up repeated builds on the same machine.

77. How can a CI runner reuse Docker build cache when every job starts on a fresh machine? ​

The short answer

Save the cache somewhere outside the machine at the end of each build, and load it at the start of the next. BuildKit can export cache to a registry, to the CI system's own cache storage, or to cloud storage, with --cache-to, and read it back with --cache-from. Another option is to stop using fresh machines for builds, by keeping a long-lived remote builder that all jobs share.

A hosted CI runner is usually a brand-new virtual machine for every job. Its build cache is empty. Every build starts from nothing, so a five-minute build takes five minutes, every time.

CI job 1, fresh machinefull build: 5 minutes--cache-to registryRegistryshop/api:buildcacheCI job 2, fresh machine--cache-from registryonly changed steps: 40 sexportimportjob 2 exports the updated cache
Cache that outlives the machine. Each job pulls the cache, reuses every unchanged step, builds the rest, and pushes the cache back for the next job.

Option 1: cache in the registry. Works on any CI system.

bash
docker buildx build \
  --cache-from type=registry,ref=myacr.azurecr.io/shop/checkout-api:buildcache \
  --cache-to type=registry,ref=myacr.azurecr.io/shop/checkout-api:buildcache,mode=max \
  -t myacr.azurecr.io/shop/checkout-api:3f9c2ab --push .

--cache-from reads the cache stored under a special tag. --cache-to writes the updated cache back. mode=max saves the cache of every stage, not only the final one, which matters for multi-stage builds. Question 78 explains the modes.

Option 2: the CI system's own cache. GitHub Actions has a built-in cache service, and BuildKit can write straight to it:

yaml
- uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v6
  with:
    push: true
    tags: myacr.azurecr.io/shop/checkout-api:${{ github.sha }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

type=gha stores the cache in GitHub's cache storage. The storage has a size limit per repository, and old entries are removed automatically when it fills.

Option 3: a long-lived builder. Run BuildKit permanently on one strong machine, or in the cluster, and have CI jobs send their builds to it with docker buildx create --driver remote. Its local cache, including cache mounts, stays warm. The cost is a shared machine you must run and secure.

One caution: cache is a shortcut, so it must be trustworthy. Do not let builds from untrusted pull requests write to the cache that release builds read from. A poisoned cache would put an attacker's layer into a release. Question 223 covers this.

78. What is the difference between inline cache, registry cache, and a local build cache? ​

The short answer

Inline cache stores cache information inside the image you push, so the image itself can serve as cache. It is simple but only covers the final stage. Registry cache stores the cache as a separate object in the registry, and can cover every stage. Local cache is either the builder's own cache on its machine, or a cache exported to a folder that your CI system saves and restores.

InlineRegistryLocal
Where it livesInside the imageSeparate registry tagBuilder's disk or a folder
Covers all stagesNo, final stage onlyYes, with mode=maxYes, with mode=max
Extra uploadSmallSeparate cache uploadNone, or a folder copy
Works on a fresh machineYesYesOnly if the folder is restored
Optiontype=inlinetype=registry,ref=...type=local,dest=...

Inline cache:

bash
docker buildx build --cache-to type=inline -t shop/api:1.4 --push .
docker buildx build --cache-from shop/api:1.4 -t shop/api:1.5 --push .

The pushed image carries a little extra information about its own layers. The next build names that image as its cache source. It is the easiest to set up. Its weakness: it only records the final image's layers, so in a multi-stage build the expensive build stages are not cached at all.

Registry cache is the usual choice for multi-stage builds in CI, as shown in question 77. The cache sits under its own tag, such as :buildcache, separate from the real image, so it never gets deployed by mistake.

Local cache means two things:

  1. The builder's own cache. Every BuildKit builder keeps a cache on its own disk. This is what makes repeated builds on your laptop fast. On a fresh machine it is empty.
  2. Exporting to a folder. --cache-to type=local,dest=/tmp/buildcache writes the cache to a folder, and your CI system saves and restores that folder between jobs with its own caching feature.

Modes, in one sentence each: mode=min, the default, caches only the layers of the final image. mode=max caches the layers of every stage, which is larger but much more useful.

There are also cloud storage backends, type=gha for GitHub Actions, and type=s3 and type=azblob for cloud object storage.

79. How do you pass a private package-registry credential to a build without baking it into the image? ​

The short answer

Use a BuildKit secret mount. The build command reads the secret from a file or an environment variable on the build machine, and the RUN step that needs it gets it as a temporary file, which exists only while that step runs. It is never written into a layer, the image history or the build cache. For private Git repositories, an SSH mount does the same with your SSH agent.

Say npm ci must download packages from a private registry, which needs a token in an .npmrc file.

dockerfile
# syntax=docker/dockerfile:1
FROM node:20-slim AS build
WORKDIR /app
COPY package.json package-lock.json ./
# (1)
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci
bash
# (2)
docker build --secret id=npmrc,src=$HOME/.npmrc -t shop/api:1.4 .
# (3)
docker build --secret id=npm_token,env=NPM_TOKEN -t shop/api:1.4 .
  1. Mounts the secret named npmrc as the file /root/.npmrc, only for this step. npm reads it and authenticates. When the step ends, the file disappears. It is held in memory, never on the layer's disk.
  2. Supplies that secret from a file on the build machine.
  3. Or from an environment variable on the build machine. A step that mounts it with --mount=type=secret,id=npm_token finds it as the file /run/secrets/npm_token, the default location. With --mount=type=secret,id=npm_token,env=NPM_TOKEN instead, that one step sees it as an environment variable.

In GitHub Actions, the official build action has a secrets: input that passes a pipeline secret straight into this mechanism.

For private Git repositories, use an SSH mount:

dockerfile
RUN --mount=type=ssh git clone git@github.com:shop/private-lib.git
bash
docker build --ssh default .

--ssh default forwards your machine's SSH agent into that one step. The private key itself never enters the build at all, only the ability to use it for the length of the step.

One thing to check: the command itself must not save the secret. Some tools copy credentials into their own config files. If that file lands in the layer, the secret leaks anyway. Make sure the step writes nothing secret to disk outside the mounted path.

80. Why are Docker build secrets safer than putting credentials in ARG, ENV, or copied configuration files? ​

The short answer

Because every other way leaves a permanent copy that anyone who can pull the image can read. ENV writes the value into the image config. ARG values used in a step are written into the image history. A copied file is stored in a layer, and deleting it later only hides it. A secret mount is the only way where the value is never written to the image, its history or the build cache at all.

MethodWhere the secret ends upReadable by
ENV TOKEN=...Image config, every containerAnyone with the image
ARG TOKEN used in RUNImage historyAnyone with the image
COPY .npmrc then rmAn earlier layerAnyone who unpacks the layer
Secret mountNowhereNobody

Why "delete it afterwards" does not work. Layers only add (question 73). If you COPY .npmrc in one step and RUN rm .npmrc in the next, the first layer still contains the file. Anyone can export the image with docker save, unpack that layer, and read the token.

Why it matters so much. Images travel. They are pushed to registries, pulled by many machines, cached on laptops, and sometimes made public by mistake. A credential in an image is a credential given to everyone who ever touches that image, for as long as the image exists. Rotating the credential is the only cure once it leaks.

What secret mounts guarantee. The secret exists only in memory, only inside the one step that mounts it, and only for the duration of that step. Its value is also not used in the cache key, so changing the token does not force a rebuild. And it does not appear in any build attestation.

What they do not guarantee. If the command prints the secret to the build log, or writes it into a file that stays in the layer, it leaks through that route instead. Secret mounts protect the value from Docker, not from your own commands.

81. What are build-time and runtime dependencies, and why should they be separated? ​

The short answer

Build-time dependencies are only needed to produce the program: compilers, header files, bundlers, type checkers and linters. Runtime dependencies are needed while the program runs: the language runtime, the shared libraries it loads, and the production packages it imports. Separating them makes images smaller and safer, because nothing in the final image is there unless the running app needs it.

A Python example makes it concrete. The psycopg2 library talks to PostgreSQL. To install it from source, you need a C compiler and PostgreSQL's development files. To run it, you need only PostgreSQL's client library.

PurposeBuild-time packageRuntime package
C compilergcc, build-essentialNot needed
PostgreSQLlibpq-dev, with headerslibpq5, the library only
TypeScripttypescript, tscNot needed, already compiled
LintingeslintNot needed

The -dev packages contain header files, which describe a library's functions so a compiler can build against it. Once the program is compiled, the headers are never read again.

Why separate them:

  1. Size. Compilers and development packages are often hundreds of megabytes.
  2. Security. A compiler on a production server lets an attacker build tools. Development packages add many more vulnerability findings to triage.
  3. Clarity. When the runtime image holds only runtime needs, it documents exactly what production depends on.
  4. Licensing. Some build tools come with license terms you do not want to distribute inside an image.

How to separate them: in a multi-stage build, install build dependencies only in the build stage, and install the small runtime libraries in the final stage:

dockerfile
FROM python:3.12-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends gcc libpq-dev && rm -rf /var/lib/apt/lists/*
RUN pip wheel --no-cache-dir --wheel-dir /wheels psycopg2==2.9.9

FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends libpq5 && rm -rf /var/lib/apt/lists/*
COPY --from=build /wheels /wheels
RUN pip install --no-cache-dir /wheels/* && rm -rf /wheels

The first stage compiles the library into a wheel, Python's ready-to-install package file. The final stage installs that wheel with only libpq5 present.

82. How would you build a multi-platform image for linux/amd64 and linux/arm64? ​

The short answer

Use Buildx with a builder that supports several platforms, and pass --platform linux/amd64,linux/arm64. BuildKit builds the image once for each platform, using emulation, native machines or cross-compilation, and pushes them together under one tag as an image index. Each machine that pulls the tag automatically gets the version for its own processor.

Intel and AMD servers use the amd64 instruction set. Apple's M-series laptops, AWS Graviton and Azure Cobalt servers use arm64. A program compiled for one cannot run on the other. If you push an amd64-only image and a team runs it on an ARM cluster, every pod fails with exec format error.

bash
docker buildx create --name multi --driver docker-container --use                 # (1)
docker run --privileged --rm tonistiigi/binfmt --install arm64,amd64             # (2)
docker buildx build --platform linux/amd64,linux/arm64 \
  -t myacr.azurecr.io/shop/checkout-api:1.4 --push .                              # (3)
docker buildx imagetools inspect myacr.azurecr.io/shop/checkout-api:1.4          # (4)
  1. Creates a builder that runs in its own container. The default Docker builder can only build multi-platform images if Docker is set up to use containerd's image store, so a separate builder is the dependable choice.
  2. Installs QEMU emulators into the machine's kernel. QEMU is a program that lets a computer run code compiled for a different processor by translating each instruction. This is only needed when you build a platform the machine cannot run natively. Question 83 compares the alternatives.
  3. Builds both platforms and pushes them. --push is needed because a multi-platform result is a set of images, which a simple local image store cannot hold.
  4. Shows what was pushed:
Name:      myacr.azurecr.io/shop/checkout-api:1.4
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:41c9...
Manifests:
  Name:      myacr.azurecr.io/shop/checkout-api:1.4@sha256:8a2e...
  Platform:  linux/amd64
  Name:      myacr.azurecr.io/shop/checkout-api:1.4@sha256:c17b...
  Platform:  linux/arm64

The tag points at an image index: one digest for the index, and one manifest per platform. A pulling machine reads the index and picks its own entry (question 35).

Inside the Dockerfile, BuildKit gives each platform's build some automatic variables. TARGETPLATFORM, TARGETOS and TARGETARCH describe the platform being built, such as linux/arm64. BUILDPLATFORM describes the machine doing the building. Question 83 uses them for fast cross-compiling.

83. What are the trade-offs between QEMU emulation, native builders, and cross-compilation for multi-platform builds? ​

The short answer

QEMU emulation needs no changes to the Dockerfile, but it is slow, often five to twenty times slower for heavy compiling, and occasionally hits emulation bugs. Native builders run each platform's build on a real machine of that type, which is fast and accurate but means running more machines. Cross-compilation compiles on the build machine for the other platform, which is the fastest of all, but needs a language and Dockerfile that support it.

QEMU emulationNative buildersCross-compilation
SpeedSlowFastFastest
Dockerfile changesNoneNoneYes
Extra machinesNoneOne per platformNone
AccuracyRare emulation bugsExactExact if done right
Best forSmall images, occasional buildsBusy pipelinesGo, Rust, Java

QEMU emulation translates every instruction of the foreign platform. An npm ci that compiles native modules, or a large C++ build, can take ten times longer under emulation. It is perfect for light images and occasional builds.

Native builders. Buildx can combine machines of different types into one builder:

bash
docker buildx create --name multi --node amd-1 ssh://ci@amd-builder
docker buildx create --name multi --append --node arm-1 ssh://ci@arm-builder

Each platform's build runs on a real machine of that type. CI providers now offer ARM runners, which makes this much easier than it used to be.

Cross-compilation runs the compiler on the build machine's own platform and tells it to produce code for the target platform. With Go it is two variables:

dockerfile
# (1)
FROM --platform=$BUILDPLATFORM golang:1.23 AS build
ARG TARGETOS TARGETARCH
WORKDIR /src
COPY . .
# (2)
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o /out/app ./cmd/app
# (3)
FROM gcr.io/distroless/static-debian12
COPY --from=build /out/app /app
ENTRYPOINT ["/app"]
  1. --platform=$BUILDPLATFORM runs the build stage on the build machine's own processor, natively, with no emulation, for every target.
  2. Go is told which operating system and processor to compile for. The compile runs at full speed and produces an ARM binary on an Intel machine, or the other way round.
  3. The final stage has no --platform, so it uses the target platform's base image. Copying a file into it needs no emulation.

Java is a special case. Java bytecode runs on any processor, so the JAR is built once. Only the base image differs per platform, and official Java images are already published for both.

84. How would you diagnose an image that builds successfully on a developer's laptop but fails in CI? ​

The short answer

Look for what is different between the two machines. Compare the processor type, the state of the build cache and the base image version. Compare the files present in the build context, and the network and credentials. Finally, compare the Docker version and the operating system's file rules. The fastest way to find it is to reproduce CI's conditions on the laptop: a clean clone, no cache, a fresh base image pull and CI's platform.

The usual differences, roughly in order of how often they bite:

  1. Processor type. An Apple laptop builds arm64. The CI machine builds amd64. A package with no ARM or no Intel version, or a step that downloads a binary for one type, works on one and fails on the other.
  2. The cache hides a broken step. The laptop's cache still holds a step that succeeded months ago. CI builds from nothing and runs that step for real, which now fails, for example because a package version was removed from the server.
  3. A different base image. The laptop has an old local copy of node:20. CI pulls the current one, which has changed.
  4. Files that exist only on the laptop. A generated file, a local .env, or a file listed in .gitignore is in the laptop's folder but not in the Git checkout CI uses. COPY fails in CI with "file not found".
  5. Network and credentials. CI cannot reach a private registry, sits behind a proxy, or does not have the secret the laptop gets from the developer's own login.
  6. Letter case in file names. macOS and Windows treat Config.js and config.js as the same file by default. Linux treats them as different files. A wrong-case import works on the laptop and fails in CI.
  7. Line endings. A shell script saved on Windows has CRLF line endings. Linux reads the hidden \r as part of the command, and the error is /bin/sh^M: bad interpreter.
  8. Lost permissions. A script that is executable on the laptop loses its executable flag in Git, and fails in CI with "permission denied".
  9. Resources. CI machines often have less memory, so a heavy compile is killed for running out of memory.

How to reproduce CI locally:

bash
git clone --depth 1 https://github.com/shop/checkout-api /tmp/clean && cd /tmp/clean   # (1)
docker build --no-cache --pull --platform linux/amd64 --progress=plain -t test .       # (2)
  1. A fresh clone contains only what is committed, exactly like CI's checkout.
  2. No cache, a fresh base image, CI's processor type, and full output. If this fails, you have reproduced the CI failure and can fix it on your laptop.

85. How would you make a Docker build deterministic and reproducible? ​

The short answer

Fix every input and remove every source of randomness. Pin the base image by digest, pin every dependency with lock files, pin system package versions, and verify every download with a checksum. Then stop the build from adding its own variation: fixed timestamps through SOURCE_DATE_EPOCH, no build dates or random values in files, and stable compiler flags. Prove it by building twice and comparing the digests.

There are two levels:

  • Repeatable: building the same commit twice gives images that behave the same.
  • Bit-for-bit reproducible: building the same commit twice gives images with the same digest. This is the stronger goal, because then anyone can rebuild an image and prove it matches what was shipped.

Step 1: fix the inputs.

  1. Base image by digest. FROM node:20-slim@sha256:... always starts from the same bytes (question 86).
  2. Language dependencies by lock file. package-lock.json, poetry.lock, go.sum, used with exact-install commands such as npm ci.
  3. System packages by version. apt-get install -y curl=7.88.1-10+deb12u8. For full reproducibility, point apt at a dated snapshot of the package archive, such as snapshot.debian.org, so the same versions are always available.
  4. Downloads by checksum. ADD --checksum=sha256:..., or curl followed by sha256sum -c.

Step 2: remove the build's own variation.

  1. Timestamps. Every file in a layer carries a modification time. If each build stamps the current time, the layer bytes differ and so does the digest. Set the SOURCE_DATE_EPOCH variable to a fixed point in time, usually the commit's timestamp. Recent BuildKit versions use it for image metadata, and with the image exporter's rewrite-timestamp=true option they also reset the times of files in the layers.
  2. No "built at" values. A step like RUN date > /build-time makes every build unique. Use the commit's time or the commit ID instead.
  3. Stable compiler output. For example, Go's -trimpath removes the build machine's folder paths from the binary.
bash
export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)          # (1)
docker buildx build --output type=image,name=shop/api:test,rewrite-timestamp=true .   # (2)
  1. Uses the last commit's time, in seconds since 1970, as the fixed timestamp.
  2. Builds with file timestamps rewritten to that value.

Step 3: prove it. Build the same commit twice, on two machines with empty caches, and compare the digests. If they differ, compare the two images layer by layer to find what changed.

86. When should you pin a base image by digest rather than relying on a mutable tag? ​

The short answer

For production builds. A digest guarantees you build on exactly the bytes you reviewed and tested, while a tag can be moved at any time, by accident or by an attacker. The cost is that a pinned digest never picks up security fixes by itself, so pair it with an automated tool, such as Renovate or Dependabot, that opens a pull request whenever a newer digest is published.

dockerfile
FROM node:20.18.0-slim@sha256:9a1f8e2b0c4d...

Keeping the tag next to the digest is good practice. Docker uses the digest. The tag tells humans which version it is.

Why pin:

  1. Reproducibility. Today's build and next month's rebuild start from the same bytes (question 85).
  2. No surprise changes. If the image publisher pushes a broken update under the same tag, your build is not affected until you choose to update.
  3. Supply-chain safety. If a tag is ever hijacked and pointed at a malicious image, a digest pin refuses it, because the bytes no longer match.
  4. Clear reviews. Updating the base image becomes a visible change in a pull request, with its own checks.

The cost, and how to pay it. A pinned digest freezes you on that version, including its vulnerabilities. Base images get security fixes often. So pinning without automation slowly turns into running old, vulnerable software. The answer is a bot:

  • Renovate or Dependabot check regularly for a newer digest behind the same tag, and open a pull request that updates the pin.
  • The pull request runs the full pipeline, so you find out whether the update breaks anything before merging.

When a plain tag is fine: local development, quick experiments, and throwaway build tools. Anything that ships to production should be pinned and updated by automation.

87. What risks come with using latest as a production image tag? ​

The short answer

latest is just the default tag name. It does not mean newest, and it can point at anything. In production, it means you cannot tell which version is running. Different machines can run different versions under the same name. A rollback has nothing to roll back to. And a deployment may not even happen, because the configuration did not change. Use unique, never-moved tags such as the commit ID, and deploy by digest.

The risks, one by one:

  1. "Latest" is only a name. Docker uses latest when you give no tag. Nothing keeps it pointing at the newest build. Someone pushing an old branch can overwrite it.
  2. You cannot tell what is running. "Production runs checkout-api:latest" tells you nothing. Which commit? Which build? You have to look up the digest on every machine.
  3. Machines disagree. One server pulled latest yesterday and another pulled it today. Both say latest, and they run different code. Bugs then appear on "some servers but not others".
  4. Rollback is impossible. The previous latest was overwritten. To go back, you must find the old digest somewhere, under pressure, during an incident.
  5. Deployments silently do nothing. If your Kubernetes manifest says image: checkout-api:latest and you push a new latest, the manifest did not change. Kubernetes sees nothing to update, so no rollout happens. Pods restarted later pull the new image, and some old pods keep running the old one.
  6. Extra pulls. Kubernetes treats :latest specially and defaults to pulling it on every container start. That adds a registry dependency to every restart (question 353).

The fix:

  1. Tag every build with something unique: the commit ID, such as checkout-api:3f9c2ab, and a release version like 1.4.2 for releases.
  2. Deploy by digest, or record the digest alongside the tag.
  3. Turn on tag immutability in the registry. Most registries, including Azure Container Registry, ECR and Harbor, can refuse any push that would move an existing tag.

88. How do you verify which files and commands contributed most to an image's size? ​

The short answer

Start with docker history, which lists each layer with its size and the instruction that created it. That usually points straight at the guilty step. To see inside a layer, use a tool called dive, which shows every file each layer added, changed or hid, and flags wasted space, such as files added and later deleted.

Step 1: which step?

bash
docker history --no-trunc --format '{{.Size}}\t{{.CreatedBy}}' shop/api:1.4 | sort -h -r | head
612MB   RUN /bin/sh -c apt-get install -y build-essential python3-dev
180MB   RUN /bin/sh -c npm ci
 74MB   # debian.sh --arch 'amd64' out/ 'bookworm' '@1730000000'
  2MB   COPY dist/ ./dist/ # buildkit

The output is sorted largest first. Here a compiler toolchain, 612 MB, ended up in production. The fix is a multi-stage build (question 65).

Step 2: which files? dive is a free tool that opens an image and lets you browse each layer's files:

bash
dive shop/api:1.4

For each layer it shows the files that were added, changed or deleted. It also reports an "efficiency" score and lists wasted space: bytes stored in one layer and then deleted or overwritten in a later one, exactly the trap from question 73. You can run it in CI with dive --ci to fail a build whose efficiency drops below a limit.

Step 3: confirm from inside, if the image has a shell:

bash
docker run --rm --entrypoint sh shop/api:1.4 -c 'du -sh /usr/* /app/* 2>/dev/null | sort -h | tail'

This lists the largest folders. node_modules with development packages, /usr/lib full of compilers, and forgotten test data are the usual finds.

Common culprits: build tools left in the final image, development dependencies, package manager caches, .git folders copied by COPY . ., duplicated files from RUN chown, and large files deleted in a later layer.

89. How would you write a Dockerfile for a service that needs fast rebuilds but also a small final image? ​

The short answer

Combine the techniques. Use a multi-stage build, so build tools stay out of the final image. Copy dependency files before source code, and use BuildKit cache mounts for package downloads. Keep a tight .dockerignore, and use COPY --link for the final copies. Finish with a slim or distroless runtime base, a non-root user, and an exec-form start command. In CI, add registry cache export so fresh machines rebuild only what changed.

Here is a complete file for the shop's TypeScript API, with each technique marked.

dockerfile
# syntax=docker/dockerfile:1
# (1)
FROM node:20-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
# (2)
RUN --mount=type=cache,target=/root/.npm npm ci

FROM deps AS build
COPY tsconfig.json ./
COPY src/ ./src/
# (3)
RUN npm run build

FROM node:20-slim AS prod-deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev

# (4)
FROM gcr.io/distroless/nodejs20-debian12:nonroot
WORKDIR /app
ENV NODE_ENV=production
# (5)
COPY --link --from=prod-deps /app/node_modules ./node_modules
COPY --link --from=build /app/dist ./dist
EXPOSE 8080
# (6)
CMD ["dist/server.js"]
  1. The deps stage installs every dependency once. The build stage reuses it with FROM deps.
  2. A cache mount keeps npm's downloads between builds, so even a changed lock file reinstalls quickly.
  3. Source code is copied last, so a code change reuses the cached install above it and only recompiles.
  4. The final stage is distroless with Node.js: no shell, no package manager, and an unprivileged user by default.
  5. --link makes these layers independent of the base image, so a base image update reuses them without copying them again (question 468).
  6. The distroless Node.js image's entrypoint is already node, so CMD gives only the script. It is exec form, so Node.js is process 1 and receives stop signals.

With a .dockerignore that excludes node_modules, dist, .git and .env*, a code-only change typically rebuilds in a few seconds, and the final image is around 150 MB instead of over 1 GB.

In CI, add the registry cache from question 77, so a brand-new runner gets the same fast rebuild:

bash
docker buildx build \
  --cache-from type=registry,ref=myacr.azurecr.io/shop/checkout-api:buildcache \
  --cache-to type=registry,ref=myacr.azurecr.io/shop/checkout-api:buildcache,mode=max \
  -t myacr.azurecr.io/shop/checkout-api:3f9c2ab --push .

469. How do you build container images inside CI or Kubernetes without a Docker daemon? ​

The short answer

You have five common options. Mounting the host's Docker socket is the easiest and the most dangerous, because it hands out root on that machine. Docker-in-Docker runs a separate daemon in a privileged container, which isolates builds from the host's daemon but still needs privileged mode. Rootless BuildKit and Buildah build images without root and without a daemon of the host. And tools like Jib and ko build images for Java and Go without a Dockerfile at all.

When CI jobs run as pods in a Kubernetes cluster, there is usually no Docker daemon available. Modern clusters run containerd, and you should not expose it to build jobs anyway.

OptionNeeds privilegedIsolationNotes
Mount /var/run/docker.sockNo, but gives rootNoneEvery job controls the host
Docker-in-DockerYesMediumSeparate daemon per job
Rootless BuildKitNoGoodSame engine as Docker
BuildahNoGoodDaemonless, from the Podman family
Jib, koNoGoodJava and Go only, no Dockerfile

Mounting the host socket. The job talks to the host's own Docker daemon. Anything the job runs can start a privileged container and take over the host, and it can see and stop every other job's containers. Question 140 explains this danger fully. Avoid it for anything shared.

Docker-in-Docker. A docker:dind container runs its own daemon inside the job. Jobs are separated from each other, but the dind container must run in privileged mode, which removes most container isolation (question 141).

Rootless BuildKit runs buildkitd as an ordinary user, inside a normal pod, using user namespaces (question 142). You send builds to it with buildctl or docker buildx using the remote or kubernetes driver. You get every BuildKit feature, including secrets and cache export.

Buildah builds OCI images from a Dockerfile, or from shell commands, without any daemon and without root. It is the usual choice in Red Hat and Podman setups.

Jib and ko skip the Dockerfile. Jib builds Java images straight from Maven or Gradle, and ko builds Go images. Both put the app's files into well-structured layers on top of a base image and push to the registry without running any container.

A note on Kaniko. Kaniko, which builds images inside an unprivileged container, was very popular for this job. Google archived the original project in June 2025, so it no longer receives updates there. Teams using it should move to rootless BuildKit or Buildah, or to a maintained fork.

Managed build services are another way to avoid the problem completely: Azure Container Registry Tasks, Google Cloud Build and AWS CodeBuild build images on the provider's machines.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet