Skip to content

Interview PrepDocker

Processes, Signals and Restarts ​

103. What is PID 1 inside a container, and why does it matter for signal handling and zombie processes? ​

The short answer

PID 1 is the first process in the container's process namespace, normally your app. The kernel treats PID 1 specially in two ways. First, signals like SIGTERM do nothing to PID 1 unless the program has set up a handler for them, so an app that ignores signals cannot be stopped politely. Second, PID 1 must collect finished child processes, or they stay behind as zombies. And when PID 1 exits, the whole container stops.

On a normal Linux machine, PID 1 is an init system such as systemd, a program written for that job. In a container, PID 1 is whatever the image starts: usually an app that was never written to be an init system.

Special rule 1: no default signal actions. For an ordinary process, if it has not set up a handler for SIGTERM, the kernel applies the default action, which is to end it. For PID 1, the kernel skips that default, to protect the system's init from being killed by accident. So:

  • An app that handles SIGTERM works fine as PID 1.
  • An app that does not handle SIGTERM simply ignores docker stop, and is killed with SIGKILL ten seconds later.

SIGKILL cannot be ignored, even by PID 1, as long as it comes from outside the container, which it does.

Special rule 2: PID 1 adopts orphans. The figure shows what happens when a parent process dies before its child.

① NormalPID 1: node appPID 7: helper scriptPID 9: worker② The script dies firstPID 1: node appPID 7: gonePID 9: orphanadoptedby PID 1When PID 9 ends, only PID 1 can collect it.If it never does, PID 9 stays a zombie.
Why PID 1 has to clean up. An orphaned process is handed to PID 1. When it finishes, its entry stays in the process table as a zombie until PID 1 asks for its exit status.

When any process ends, a small record of it stays in the kernel's process table until its parent asks for its exit status, using a system call named wait. A finished process waiting to be collected is called a zombie. Normally the parent collects it at once.

If the parent dies first, the child becomes an orphan and the kernel makes PID 1 its new parent. A real init system collects these orphans when they finish. A Node.js or Java app does not know it is supposed to. So zombies pile up. Each one holds a process number, and when the container's pids limit is reached (question 100), no new process can start at all: "fork: resource temporarily unavailable".

Special rule 3: PID 1 is the container. When PID 1 exits, the kernel kills every other process in the namespace and the container stops.

The fix for rules 1 and 2: write the app to handle SIGTERM (question 105), and if the app starts child processes, put a tiny init such as tini in front of it (question 471).

104. What happens when you run docker stop, and what happens if the process ignores the termination signal? ​

The short answer

Docker sends the stop signal, SIGTERM unless the image sets another, to PID 1 in the container, then waits, 10 seconds by default. If the process exits in that time, the stop is clean. If it ignores the signal, or is still busy when the time runs out, Docker sends SIGKILL, which ends it immediately. The container then shows exit code 137, and any work in progress is lost.

0 s1 s10 sSIGTERM sentSIGKILL sentgood app: drains, exits 0app ignoring SIGTERM keeps running...killed, exit 137
The stop sequence. The polite signal comes first and the forced one comes at the deadline. A well-behaved app uses the time in between to finish its work.

Step by step:

  1. docker stop api asks the daemon to stop the container.
  2. The daemon sends the image's STOPSIGNAL, which defaults to SIGTERM, to PID 1 in the container. Only PID 1 gets it.
  3. Docker waits for the grace period: 10 seconds by default. You can change it per command with docker stop -t 30 api, or per container at creation with docker run --stop-timeout 30.
  4. If PID 1 exits within the grace period, the stop is done. The container is exited with the app's own exit code, usually 0.
  5. If PID 1 is still running at the deadline, Docker sends SIGKILL. The kernel ends the process at once. The exit code is 137.

Why a process might "ignore" the signal:

  • It is PID 1 and has no handler for SIGTERM, so the kernel does nothing (question 103).
  • PID 1 is a shell that does not pass the signal on to the real app (question 50).
  • It handles the signal but takes longer than the grace period, for example waiting for a long request or a slow database.

What is lost with SIGKILL: requests in progress get dropped connections, buffered logs are never written, messages taken from a queue may not be acknowledged, and files being written may be left half-finished.

One more detail: a container you stop by hand is not restarted by on-failure or always restart policies. Docker knows you asked for it (question 106).

105. How do you make an application shut down gracefully when a container is stopped? ​

The short answer

Four things together. Start the app in exec form so it is PID 1 and receives the signal. Handle SIGTERM in the code: stop accepting new work, finish the work in progress, close connections, then exit. Give that a deadline shorter than the container's grace period. And make sure the grace period is long enough for your slowest normal request.

1. Receive the signal. Use exec form, CMD ["node", "dist/server.js"], or an entrypoint script that ends with exec "$@" (questions 50 and 51).

2. Handle it in the code. Here is the shop's TypeScript API:

ts
import http from 'node:http'
import { Pool } from 'pg'

const db = new Pool({ connectionString: process.env.DATABASE_URL })
const server = http.createServer(handleRequest)
server.listen(8080)

let shuttingDown = false

async function shutdown(signal: string): Promise<void> {
  if (shuttingDown) return                                   // (1)
  shuttingDown = true
  console.log(`received ${signal}, shutting down`)

  const deadline = setTimeout(() => {                        // (2)
    console.error('shutdown took too long, exiting anyway')
    process.exit(1)
  }, 8_000)
  deadline.unref()

  server.close()                                             // (3)
  server.closeIdleConnections()                              // (4)
  await new Promise<void>((resolve) => server.on('close', () => resolve()))   // (5)
  await db.end()                                             // (6)
  console.log('clean exit')
  process.exit(0)                                            // (7)
}

process.on('SIGTERM', () => void shutdown('SIGTERM'))        // (8)
process.on('SIGINT', () => void shutdown('SIGINT'))
  1. A second signal during shutdown is ignored, so the steps never run twice.
  2. A safety deadline of 8 seconds, shorter than Docker's 10-second grace period. If something hangs, the app exits on its own terms and logs why, instead of being killed silently. unref() means this timer alone does not keep the process alive.
  3. Stops accepting new connections. Requests already being handled keep going.
  4. Closes connections that are open but idle. Browsers and load balancers keep connections open for reuse, and without this they would hold the server open until the deadline.
  5. Waits until every in-flight request has finished and the server is fully closed.
  6. Closes the database connection pool, so the database is not left with abandoned connections.
  7. Exits with 0, meaning "I stopped cleanly".
  8. Registers the handler for SIGTERM, and for SIGINT, which is what Ctrl+C sends during development.

What it prints when stopped:

received SIGTERM, shutting down
clean exit

3. Other kinds of work. A queue consumer should stop fetching new messages, finish and acknowledge the current one, then exit. A scheduled job should save its progress or finish its current item.

4. In Kubernetes, one more problem. The load balancer may keep sending new requests for a few seconds after the pod has been told to stop, because the news that the pod is leaving takes time to spread. The fix is a short preStop pause before the signal, and a terminationGracePeriodSeconds long enough for both. Question 356 covers this.

106. What are Docker restart policies, and how do no, on-failure, always, and unless-stopped differ? ​

The short answer

A restart policy tells the Docker daemon whether to start a container again when it stops by itself. no never restarts it, and is the default. on-failure restarts only when it exits with an error code, optionally up to a maximum number of tries. always restarts it whenever it stops, and also when the Docker daemon starts. unless-stopped is like always, except it stays stopped after a daemon restart if you had stopped it by hand.

bash
docker run -d --restart on-failure:5 shop/worker:1.4   # (1)
docker run -d --restart unless-stopped shop/api:1.4     # (2)
docker update --restart always api                      # (3)
  1. Restart the worker if it crashes, at most 5 times.
  2. Keep the API running through crashes and machine reboots, unless an operator deliberately stopped it.
  3. Change the policy of an existing container without recreating it.
PolicyProcess exits with errorExits with 0Stopped by youDaemon or host restarts
noStays stoppedStays stoppedStays stoppedStays stopped
on-failureRestartedStays stoppedStays stoppedRestarted if it had failed
alwaysRestartedRestartedStays stopped until daemon restartRestarted
unless-stoppedRestartedRestartedStays stoppedRestarted only if it was running

How the restarts are spaced out. If a container keeps crashing, Docker does not restart it in a tight loop. It waits a little before each restart, starting at 100 milliseconds and doubling each time. Once a container stays up for at least 10 seconds, the waiting time resets. In docker ps such a container shows as Restarting (1) 2 seconds ago, where 1 is the last exit code.

Which to choose:

  • Long-running services on a single host: unless-stopped. They survive crashes and reboots, and an operator's manual stop is respected.
  • Batch jobs that should retry on failure: on-failure:3, so a real bug does not retry forever.
  • One-off commands and tests: no, with --rm.

What a restart policy cannot do is covered in question 107. It restarts the same container on the same machine. It cannot help if the machine itself dies.

107. How is Docker's restart policy different from a process supervisor or an orchestrator's self-healing behavior? ​

The short answer

A Docker restart policy restarts the same container on the same host when its process exits, and that is all. A process supervisor, such as systemd, manages processes on one machine, with dependencies and logging. An orchestrator such as Kubernetes keeps a desired state across many machines. It replaces failed containers and moves them off broken machines. It restarts ones that hang without crashing. It keeps them out of traffic until they are ready, and it rolls out new versions.

The three work at different heights:

Docker restart policyProcess supervisorOrchestrator
Example--restart alwayssystemd, supervisordKubernetes
ScopeOne container, one hostProcesses on one hostMany containers, many hosts
Reacts to a crashYesYesYes
Reacts to a hang that does not crashNoOnly with watchdog setupYes, liveness probes
Reacts to the host dyingNoNoYes, reschedules elsewhere
Removes from traffic when not readyNoNoYes, readiness probes
Keeps a number of copies runningNoNoYes
Rolling updatesNoNoYes

What each misses:

  1. The restart policy only sees exits. If the app deadlocks and stops answering, but its process is still alive, Docker sees nothing wrong. Even an unhealthy health check does not trigger a restart in plain Docker (question 109).
  2. Everything on one machine dies with the machine. If the host's disk fails, its containers, its restart policies and its supervisor all go with it.
  3. Nobody decides how many copies there should be. If you need three copies of the API and one machine dies, plain Docker cannot start a third copy somewhere else.

Supervisors inside containers are usually a sign of trouble. Running supervisord to start several programs in one container hides crashes from Docker and from the orchestrator. The usual rule is one main process per container. An exception is a tiny init such as tini, which only forwards signals and collects zombies (question 471).

The orchestrator's model is reconciliation: it constantly compares the desired state, "three healthy copies of checkout-api version 1.4", with reality, and fixes any difference. Question 332 explains how controllers do this.

108. How do container environment variables get set, and how can you inspect which variables are present? ​

The short answer

They come from three places, in order: the image's ENV lines, then whatever you pass when the container is created, with -e, --env-file, or a Compose file's environment and env_file. Values passed at run time override the image's. To see them, use docker inspect for the container's configuration, or docker exec <container> env to see what a new process inside it gets.

bash
docker run -d --name api -e LOG_LEVEL=debug -e AWS_REGION --env-file ./api.env shop/api:1.4

The command uses all three ways at once:

  1. -e LOG_LEVEL=debug sets the variable to debug, overriding any ENV LOG_LEVEL=info in the image.
  2. -e AWS_REGION, a name with no value, copies the value from your own shell at that moment. If your shell has no such variable, it is left unset.
  3. --env-file ./api.env reads one KEY=value per line from a file.

The order of precedence, lowest to highest:

  1. ENV in the image.
  2. --env-file values.
  3. -e values.

In Docker Compose, values under environment: take precedence over values from env_file:.

How to see them:

bash
docker inspect -f '{{json .Config.Env}}' api      # (1)
docker exec api env                                # (2)
docker exec api cat /proc/1/environ | tr '\0' '\n' # (3)
  1. The variables Docker started the container with, as a JSON list.
  2. The environment a new process inside the container gets. It is built from the same configuration.
  3. The environment of PID 1, the app itself, as it was when it started. The entries in this file are separated by zero bytes, so tr turns them into new lines. Use this when an entrypoint script might have changed variables before starting the app.

One limit to know. Environment variables are fixed when a process starts. Changing a variable means recreating the container. You cannot change them in a running process from outside.

And one security point. Anyone who can run docker inspect on the host sees every value in plain text. That is why secrets are better passed as files, such as a mounted secret file, than as environment variables (question 144).

109. What is a Docker health check, and how is an unhealthy container different from a stopped container? ​

The short answer

A health check is a command Docker runs inside the container at a regular interval to test whether the app really works, for example by requesting its health page. Based on the results, the container's health is starting, healthy or unhealthy. An unhealthy container is still running: its process is alive but failing the test. A stopped container has no process at all. Plain Docker only reports an unhealthy status; it does not restart the container because of it.

dockerfile
HEALTHCHECK --interval=15s --timeout=3s --start-period=30s --retries=3 \
  CMD ["node", "dist/healthcheck.js"]
  • Every 15 seconds, Docker runs node dist/healthcheck.js inside the container.
  • Exit code 0 means pass. Any other exit code, or taking longer than 3 seconds, means fail.
  • During the first 30 seconds, the start period, failures do not count, so a slow-starting app is not marked unhealthy while it boots.
  • 3 failures in a row turn the status to unhealthy. One pass turns it back to healthy.
bash
docker ps
docker inspect -f '{{json .State.Health}}' api
STATUS
Up 4 minutes (unhealthy)
json
{"Status":"unhealthy","FailingStreak":3,"Log":[{"ExitCode":1,"Output":"database unreachable"}]}

docker ps shows the status next to the uptime. docker inspect shows the last few check results, including whatever the check printed, which often tells you exactly what is wrong.

Unhealthy versus stopped:

UnhealthyStopped
Process runningYesNo
Can still answer requestsPossibly, badlyNo
Plain Docker reactionReports it onlyRestart policy may restart it
Typical causeApp is up but a dependency or part of it is brokenApp crashed or was stopped

Who acts on the health status:

  • Docker Compose can wait for a dependency to become healthy before starting another service, with depends_on and condition: service_healthy (question 125).
  • Docker Swarm replaces unhealthy containers.
  • Kubernetes ignores Docker health checks entirely. It uses liveness, readiness and startup probes defined in the pod specification (question 401).

What a good health check tests: that the app itself can serve requests. Be careful about checking every dependency. If the health check fails whenever the database is slow, every copy of the app turns unhealthy at once, and an orchestrator that restarts unhealthy containers would restart all of them together, making things worse. Question 403 explains this trap.

471. What is tini, or docker run --init, and what problem does it solve? ​

The short answer

tini is a tiny init program, a few hundred kilobytes, that runs as PID 1 and starts your app as its child. It forwards every signal it receives to the app, collects zombie processes, and exits with the app's own exit code. It fixes both PID 1 problems from question 103 without changing your app. docker run --init adds it automatically; otherwise you install it in the image.

bash
docker run --init -d shop/api:1.4      # (1)
dockerfile
# (2)
RUN apt-get update && apt-get install -y --no-install-recommends tini && rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["node", "dist/server.js"]
  1. Docker injects its bundled copy of tini as PID 1, and your image's command becomes its child.
  2. Or install it in the image, so it works everywhere, including Kubernetes, which has no --init option. The -- tells tini that everything after it is the command to run.

With tini, the process tree inside the container looks like this:

PID 1  /usr/bin/tini -- node dist/server.js
PID 7  node dist/server.js

What tini does:

  1. Forwards signals. SIGTERM arrives at tini, and tini passes it to node. Node handles it normally. Node is no longer PID 1, so the kernel's special rule for PID 1 no longer applies to it.
  2. Collects zombies. Any orphaned process adopted by PID 1 is collected by tini when it finishes, so zombies never pile up.
  3. Passes on the exit code. When node exits, tini exits with the same code, so Docker and Kubernetes see the app's real result.

When you need it: there are three cases. Your app starts child processes, such as shell commands, browsers for PDF rendering, or workers. Or your app does not handle SIGTERM, and you cannot change it. Or the start command is a script.

When you do not: a single program that handles signals itself and never starts children, such as the Node.js server in question 105. dumb-init is a similar tool that does the same job.

472. How do Docker logging drivers work, and how do you stop container logs from filling a disk? ​

The short answer

Docker captures each container's standard output and error and sends every line to a logging driver. The default driver, json-file, writes lines to a file on the host, and by default that file is never rotated, so a chatty container can fill the disk. Set max-size and max-file options, or use the local driver, which rotates and compresses by default. Other drivers send logs straight to journald, syslog, Fluentd or a cloud service.

Set rotation for one container:

bash
docker run -d --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 shop/api:1.4

This keeps at most three files of 10 MB each. When the current file reaches 10 MB, it is rotated, and the oldest is deleted. The container can never use more than about 30 MB for logs.

Better, set defaults for every container in the daemon's configuration file, /etc/docker/daemon.json:

json
{
  "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "5" }
}

The local driver stores logs in a compact format and rotates them by default. Restart the daemon for this to apply to new containers. Existing containers keep the settings they were created with.

The common drivers:

DriverWhere logs godocker logs works
json-fileA JSON file per containerYes
localA compact, rotated fileYes
journaldThe systemd journalYes
syslogA syslog serverThrough dual logging
fluentdA Fluentd collectorThrough dual logging
awslogs, gcplogsCloud log servicesThrough dual logging

Dual logging. For drivers that send logs elsewhere, newer Docker versions also keep a small local copy, so docker logs still works.

A trap that hangs apps. By default, delivering logs is blocking: if the logging driver cannot keep up, for example because a remote log server is slow, the app's next write to standard output waits. A busy app can freeze because of its logging. The option --log-opt mode=non-blocking with a max-buffer-size puts logs in a memory buffer instead, and drops lines if the buffer fills, which is usually the better trade.

In Kubernetes, the kubelet rotates container logs itself, by default at 10 MB per file with 5 files kept, and a log collector such as Fluent Bit reads them from each node.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Interview Prep
Docker, CI/CD, Kubernetes & Cloud
Your project
Docker
CI/CD
Kubernetes
Cloud
Observability
Scenarios
Other topics
Nothing picked yet