Appearance
2.2.3 — Signals & Graceful Shutdown
It is Friday afternoon, and you deploy a new version of the photo-upload service. At that moment, forty people are halfway through uploading a photo. The old copy of the service has to stop so the new one can take over. If it simply vanishes, forty uploads fail, and forty people see an error. If it is told "please finish what you are doing and then stop", every upload completes, and nobody notices the deploy at all.
That polite "please stop" is a signal. Signals are how the kernel, the terminal and other programs get the attention of a running process from the outside. This page explains what signals are, which ones matter, how a program catches them, and how to use them to shut down without losing anyone's work. That last part is a standard interview question at every level.
1. A tap on the shoulder
A running process is busy in its own world. It runs its own code, in its own memory. So how does something outside it, such as you pressing Ctrl+C, or the kernel noticing a bad memory access, or a deploy tool, interrupt it?
The answer is a signal: a small, numbered notification that the kernel delivers to a process. The signal interrupts whatever the process was doing and says "something happened". A signal carries almost no information. It is just a number, such as 15, with a name, such as SIGTERM. It is the operating system's tap on the shoulder. Some signals are a polite tap. One is a bullet.
When a signal arrives, one of three things happens:
- The default action. Every signal has a default, decided by the kernel. For most signals the default is to end the process.
- A handler runs. The process can install a signal handler, a function the kernel runs as soon as that signal arrives. The handler can clean up, save work, or decide to ignore the signal.
- The signal is ignored. The process can tell the kernel to throw that signal away.
Two signals are special: the process can neither catch nor ignore them. Section 2 explains why.
2. The signals you need to know
There are about thirty standard signals. These are the ones you will actually meet. The numbers are the ones used by Linux on a PC.
| Signal | Number | Sent when | Default | Can catch? |
|---|---|---|---|---|
SIGHUP | 1 | terminal closed | end | yes |
SIGINT | 2 | Ctrl+C | end | yes |
SIGQUIT | 3 | Ctrl+\ | end + core dump | yes |
SIGKILL | 9 | kill -9 | end | no |
SIGSEGV | 11 | bad memory access | end + core dump | yes |
SIGPIPE | 13 | wrote to a closed pipe | end | yes |
SIGTERM | 15 | kill, deploys | end | yes |
SIGCHLD | 17 | a child finished | ignore | yes |
SIGSTOP | 19 | pause request | pause | no |
SIGCONT | 18 | resume request | resume | yes |
A core dump, in the default column, is a file containing a copy of the process's memory at the moment it died. Programmers open it later in a debugger to see what went wrong.
Now the important ones, one at a time.
SIGINT means "interrupt". It is what Ctrl+C sends to the program running in your terminal. It is a polite request to stop. A program can catch it, for example to ask "do you really want to quit?", or to save your work first. ⚑What is SIGINT? [EQ-90]
SIGTERM means "terminate". It is the polite, catchable request to shut down, and it is what the kill command sends if you do not name a signal. This is the signal that matters most in production. Deploy tools, process managers and container platforms all send SIGTERM when they want a program to stop. It means "please finish up and exit cleanly". A well-built server catches it and does a graceful shutdown, which section 5 builds. ⚑What is SIGTERM? [EQ-91]
SIGKILL means "kill". It cannot be caught and cannot be ignored. When it is sent, the kernel ends the process immediately, without running a single further instruction of it. kill -9 sends it. Because the process gets no chance to react, it cannot finish writing a file, cannot release a lock it holds in a database, and cannot tell the people it is serving. That is exactly why you send SIGTERM first, and use SIGKILL only if the process ignores the polite request.
Why must SIGKILL be uncatchable? Because there has to be one way to stop a program that is stuck, broken or hostile. If every signal could be caught, a program could catch them all and refuse to ever stop. SIGKILL is the kernel's guarantee that the person in charge of the machine always wins. SIGSTOP, which pauses a process, is uncatchable for the same reason.
SIGSEGV means "segmentation violation". The kernel sends it when a process touches memory it does not own, which is the crash 2.1.2 described. The process did not ask for it. The kernel sends it because of what the process did. 2.5.3 explains the memory side.
SIGHUP means "hang up". It dates from when terminals were connected to computers by telephone lines, and the signal meant "the line dropped". Today it is sent when you close the terminal window a program is running in, which is why closing a terminal usually kills the programs started from it. 2.8.2 explains how to keep a program running after you close the terminal. Many long-running servers reuse SIGHUP with a different meaning: "reload your configuration file without restarting".
SIGCHLD is sent to a parent when one of its children finishes. Its default is to be ignored. A parent that wants to reap its children promptly (2.2.2) catches it and calls wait in the handler.
SIGPIPE is sent when a process writes into a pipe or a network connection whose other end has been closed. Page 2.2.4 explains pipes, and shows how Node handles this one.
⚑What signals can a process listen to? [EQ-89]Sending a signal yourself
On Linux and macOS, kill sends signals. Its name is misleading, because it can send any signal, not only deadly ones.
sh
kill 4012 # send SIGTERM (15), the polite default, to process 4012
kill -TERM 4012 # the same thing, with the name spelled out
kill -9 4012 # send SIGKILL: end it now, no cleanup
kill -HUP 4012 # send SIGHUP: many servers reload their configuration
kill -0 4012 # send nothing; only check that process 4012 existskill -0 is a handy trick. Signal number 0 does not exist, so nothing is delivered, but the kernel still checks that the process exists and that you are allowed to signal it. Scripts use it to ask "is that process still alive?".
3. How the kernel delivers a signal
Knowing the mechanism explains several surprising behaviours, so here it is.
- Something asks the kernel to send a signal. That might be a
killsystem call from another process, the terminal when you press Ctrl+C, or the kernel itself after a bad memory access. - The kernel marks the signal as pending. In the target process's control block (2.2.1), the kernel sets a bit that means "signal 15 is waiting". Nothing else happens yet.
- The kernel delivers it at the next chance. The next time the target process is about to return from kernel mode to user mode, after a system call, after an interrupt, or when it is next scheduled, the kernel checks the pending bits.
- The kernel acts. If the process has a handler for that signal, the kernel arranges for the handler to run before the process continues its normal code. When the handler returns, the process carries on exactly where it was interrupted. If there is no handler, the kernel applies the default action.
Two consequences follow from this, and both catch people out.
A process can block signals for a while. A process can tell the kernel "do not deliver SIGINT to me right now", using a set of bits called the signal mask. A blocked signal is not lost. It stays pending until the process unblocks it, and then it is delivered. Programs do this around short pieces of code that must not be interrupted halfway.
Standard signals do not queue up. "Pending" is a single bit per signal, not a counter. If three children finish at almost the same moment, the parent may receive just one SIGCHLD. So a handler must never assume one signal means one event. A correct SIGCHLD handler calls wait in a loop until no more finished children are left.
4. Why signal handlers are dangerous, and why Node's are not
A signal handler runs at a moment the program did not choose. It interrupts the main code at any instruction. Suppose the main code was in the middle of malloc, the C function that hands out memory, and malloc was holding its internal lock. If the handler now calls malloc too, the handler waits for the lock. But the lock is held by the very code the handler interrupted, which cannot continue until the handler returns. The program freezes forever.
Because of this, C handlers may only call functions on a short list that are known to be safe at any moment. These are called async-signal-safe functions. printf and malloc are not on the list. The safe habit in C is for a handler to do almost nothing: set a flag variable, and return. The program's main loop checks the flag at a safe moment and does the real work there.
Node does this for you. When you write process.on("SIGTERM", handler) in TypeScript, Node does not run your function inside the real, low-level signal handler. Node installs its own tiny C handler, which only writes a byte to wake up the event loop (3.8.1). Your TypeScript function then runs later, as an ordinary event, at a safe moment between other events. So in Node you can log, close servers and open files in a signal handler without any of the danger above. It is still worth knowing the danger exists, because it is a favourite question in systems interviews.
5. Graceful shutdown, built step by step
Here is the problem from the top of the page, solved. A container platform or a deploy tool wants the old copy of the upload service to stop. It does not just pull the plug. It follows a fixed sequence:
- It sends SIGTERM.
- It waits a grace period. On Kubernetes the default is 30 seconds (14.3.5).
- If the process is still running when the period ends, it sends SIGKILL.
So the service has up to 30 seconds to finish its work. A graceful shutdown uses that time well. In order, the service must:
- Stop accepting new uploads.
- Finish the uploads already in progress.
- Close what it holds open, such as database connections, and flush anything still in memory to disk.
- Exit with code 0, before the deadline.
Here is the upload service doing exactly that.
ts
// graceful-shutdown.ts — the upload service finishing its work before it exits
import { createServer, request } from "node:http";
let inFlight = 0; // uploads being handled right now
let shuttingDown = false;
const server = createServer((req, res) => {
inFlight++;
setTimeout(() => { // pretend saving the photo takes 2 s
if (shuttingDown) res.setHeader("Connection", "close"); // (1)
res.end("saved\n");
inFlight--;
console.log("one upload finished");
}, 2000);
});
function shutdown(signal: string): void {
if (shuttingDown) return; // (2)
shuttingDown = true;
console.log(`${signal} received, ${inFlight} upload(s) in flight`);
server.close(() => { // (3)
console.log("all uploads finished, exiting with code 0");
process.exit(0); // (4)
});
setTimeout(() => { // (5)
console.log("deadline passed, exiting with code 1");
process.exit(1);
}, 25_000).unref(); // (6)
}
process.on("SIGTERM", () => shutdown("SIGTERM")); // (7)
process.on("SIGINT", () => shutdown("SIGINT"));
server.listen(8080, () => {
// Test drive: start one slow upload, then ask the service to stop.
request("http://localhost:8080/upload").end();
setTimeout(() => process.emit("SIGTERM"), 200); // stands in for `kill <pid>`
});
// Prints, and the process exits with code 0 after about 2.2 seconds:
// SIGTERM received, 1 upload(s) in flight
// one upload finished
// all uploads finished, exiting with code 0Walk through the marked lines:
(7)We register a handler forSIGTERM, and the same one forSIGINT, so Ctrl+C during development behaves like a deploy. Without these lines,SIGTERM's default action would end the process instantly, in the middle of the upload.(2)A shutdown must run only once. A deploy tool may sendSIGTERMtwice, and a person may press Ctrl+C twice. The flag makes the second signal harmless.(3)server.closedoes two things. It stops accepting new connections at once, which is step 1. And it calls our function only when every existing connection has ended, which covers step 2. While we wait, the in-flight upload carries on normally.(1)This line looks small, but it decides whether the shutdown takes 2 seconds or 7. Browsers and other clients keep a connection open after a response, ready for their next request. This is called keep-alive. An open idle connection would stopserver.closefrom finishing until it timed out on its own, which takes 5 seconds by default in Node. So while shutting down, every response tells the client "close this connection". When we tried the same program without this line, the shutdown took 6.2 seconds instead of 2.2.(4)Everything is finished, so we exit with code 0, which means "success". This is where a real service would also close its database connections, which is step 3.(5)A safety net. If something hangs, for example an upload whose client vanished halfway, we must not wait forever. We pick a deadline shorter than the platform's grace period, here 25 seconds against Kubernetes's 30, so we decide how to exit rather than being killed.(6)unref()tells Node "this timer alone is not a reason to stay alive". Without it, the 25-second timer itself would keep the process running after everything else had finished.
The last line, process.emit("SIGTERM"), is only for testing. It runs our handler directly, as if the signal had arrived. That is how the output above was produced, on Windows, which cannot deliver a real SIGTERM from another process to a Node program. On Linux and macOS you would start the service and run kill <pid> from another terminal, and the output is the same.
This is the operating-system half of the story. A real web service adds two more things. First, it tells the load balancer to stop sending it traffic before it closes, using a readiness check. Second, it handles requests that use streaming or long-lived connections. 9.9.7 builds the complete Express version. ⚑How do you gracefully shut down an application, and which signals are involved? [EQ-88]
6. Exit codes tell you how a process died
When a process ends, its parent learns how it ended (2.2.2). If it exited by itself, the parent gets the exit code it chose, from 0 to 255. If a signal killed it, shells and container platforms report the code as 128 plus the signal number. That one rule lets you read a crash from a single number.
| Exit code | Meaning |
|---|---|
| 0 | finished successfully |
| 1, 2, … | the program reported its own error |
| 127 | command not found |
| 130 | killed by SIGINT (128 + 2), Ctrl+C |
| 137 | killed by SIGKILL (128 + 9) |
| 139 | killed by SIGSEGV (128 + 11), a memory crash |
| 143 | killed by SIGTERM (128 + 15) |
Code 137 is the one to memorise. It means the process was killed by SIGKILL, and in production there are two usual reasons:
- It ignored
SIGTERM, and the grace period ran out. - It used more memory than its limit, and the kernel's out-of-memory killer ended it. 2.5.3 explains the out-of-memory killer, and on Kubernetes the pod's status says
OOMKilled.
Code 143 is the healthy version of the same story. It means the process received SIGTERM and simply let the default action end it. It did stop when asked, but it did no cleanup. A service with a proper handler, like ours, exits with 0 instead.
7. The container trap: PID 1 ignores signals
Here is a bug that has wasted countless hours. You put the upload service in a container. You write the perfect SIGTERM handler. You deploy. And every shutdown still takes exactly 30 seconds and ends with code 137. The handler never ran. Why?
There are two separate causes, and either one is enough.
Cause 1: a shell is PID 1, not your program. If the container is told to start the service through a shell, for example with the "shell form" of a Dockerfile's start command, then the container's first process is /bin/sh, and it starts node as its child. The SIGTERM goes to PID 1, which is the shell. The shell does not pass it on to its child. So node never hears about it, and after 30 seconds everything is killed. The fix is to start node directly, so that it is PID 1 itself. In a Dockerfile that is the "exec form", written as a list: CMD ["node", "upload-service.js"] instead of CMD node upload-service.js. 14.2.3 covers the two forms.
Cause 2: PID 1 gets special treatment from the kernel. The kernel protects PID 1, because if PID 1 died, the whole system (or the whole container) would go with it. So the kernel does not apply default actions to PID 1. For an ordinary process, an uncaught SIGTERM ends it. For PID 1, an uncaught SIGTERM is simply ignored. Only signals that PID 1 has explicitly installed a handler for have any effect. A Node program running as PID 1 with a SIGTERM handler, like ours, works correctly. But a program without one, or a tool that relies on the default, ignores the signal and gets killed 30 seconds later.
Both problems, and the zombie problem from 2.2.2, have the same clean fix: run a tiny init program as PID 1. Programs such as tini, or Docker's built-in --init option, are built for exactly this job. The init program runs as PID 1. It starts your service as its child. It passes every signal it receives on to that child. And it reaps every orphan it adopts. Your service then runs as an ordinary process, with ordinary signal behaviour.
What the interviewer will push on
"How do you shut down a service gracefully?" This is the core question, and they are checking whether you know the sequence, not just the word. A good answer: the platform sends SIGTERM, waits a grace period, then sends SIGKILL. The service catches SIGTERM, stops accepting new work, finishes in-flight work, closes connections, and exits before the deadline. The tell of real experience is mentioning the keep-alive connections that hold server.close open, or the readiness check that stops new traffic arriving.
"What is the difference between SIGTERM and SIGKILL?" SIGTERM can be caught and asks politely. SIGKILL cannot be caught and ends the process at once with no cleanup. The strong answer says why SIGKILL must be uncatchable: there must always be a way to stop a stuck or hostile program.
"A container exits with code 137. What happened?" They want 128 + 9 = SIGKILL, and the two usual causes: it ignored SIGTERM until the grace period ended, or it hit its memory limit and the out-of-memory killer ended it.
"My SIGTERM handler works locally but not in the container. Why?" This is the senior follow-up. Name both causes: a shell as PID 1 that does not forward the signal, and the kernel not applying default actions to PID 1. The fix is the exec form, or tini and --init.
"What can you safely do inside a signal handler?" In C, almost nothing: set a flag and return, because the handler may interrupt malloc or a held lock. In Node, anything, because Node runs your handler later as a normal event.
Something to volunteer that nobody asks for: standard signals do not queue. Three children finishing together may produce one SIGCHLD, so a handler must reap in a loop. Mentioning this shows you have read signal code that actually broke.
Recall
- A signal is a numbered notification the kernel delivers to a process. It triggers the default action, a signal handler, or nothing if ignored.
- SIGINT (2) is Ctrl+C. SIGTERM (15) is the polite, catchable "please shut down" sent by
killand by deploys. SIGKILL (9) cannot be caught or ignored and ends the process with no cleanup. SIGSEGV (11) is a bad memory access. SIGHUP (1) is a closed terminal, or "reload config". - Signals are marked pending in the process's control block and delivered on the next return to user mode. They can be blocked temporarily, and they do not queue.
- C handlers may only call async-signal-safe functions. Node runs your handler later as a normal event, so it is safe.
- A graceful shutdown: catch
SIGTERM, stop accepting work, finish in-flight work, close connections, exit 0 before the grace period ends. Exit code 128 + signal: 137 isSIGKILL, 143 isSIGTERM. In a container, PID 1 ignores signals it has no handler for, and a shell as PID 1 does not forward them; use the exec form ortini.
Self-test: Why must SIGKILL be impossible to catch? What four things does a graceful shutdown do, and in what order? Why did our shutdown take 6 seconds without the Connection: close line? A pod exits with 137. What are the two likely causes? Why does a SIGTERM handler that works on a laptop sometimes never run in a container?
Quiz Bank
FoundationalWhat is a signal, and what can a process do when it receives one?
A signal is a small numbered notification that the kernel delivers to a process to tell it something happened, such as Ctrl+C being pressed, a request to shut down, or a bad memory access. It interrupts the process's normal flow.
The process can let the signal's default action happen, which for most signals ends the process. It can install a signal handler, a function the kernel runs when the signal arrives. Or it can ignore the signal. The exceptions are SIGKILL and SIGSTOP, which can be neither caught nor ignored.
FoundationalWhat is the difference between SIGTERM and SIGKILL?
SIGTERM (15) is a polite request to shut down. The process can catch it, finish its work, clean up and exit. It is what kill sends by default and what deploy tools and container platforms send first.
SIGKILL (9) cannot be caught or ignored. The kernel ends the process at once, without running any more of its code, so it cannot save files, release locks or finish requests. It exists so that a stuck or hostile program can always be stopped. The right order is always SIGTERM first, then SIGKILL only if the process has not exited after a grace period.
AppliedWhy do C programs keep signal handlers tiny, and why does that rule not apply to Node handlers?
A C signal handler can run at any instruction of the main program, including in the middle of malloc while it holds an internal lock. If the handler calls malloc or printf too, it may wait for a lock held by the code it interrupted, and the program freezes. So C handlers may only call async-signal-safe functions, and the usual habit is to set a flag and return, letting the main loop do the real work.
Node installs its own tiny C handler that only wakes the event loop. Your TypeScript handler then runs later as an ordinary event, between other events, at a safe moment. So in Node a handler can log, close servers and use files freely.
InterviewHow does graceful shutdown work, and which signals are involved?
When a platform wants to stop a process, for a deploy or a scale-down, it sends SIGTERM and waits a grace period, 30 seconds by default on Kubernetes. A well-built service catches SIGTERM and:
- stops accepting new connections;
- finishes the requests already in progress, telling keep-alive clients to close their connections;
- closes database and queue connections and flushes buffers;
- exits with code 0, before its own deadline, which should be shorter than the grace period.
If the process is still alive when the grace period ends, the platform sends SIGKILL, which cannot be caught, and in-flight work is lost. Ctrl+C sends SIGINT, so services usually handle it the same way for development.
InterviewA container exits with code 137. What does that tell you?
Exit codes above 128 mean the process was killed by a signal, and the signal number is the code minus 128. 137 − 128 = 9, which is SIGKILL. Nothing could catch it, so the process did no cleanup.
In production there are two usual causes. Either the process ignored or did not finish handling SIGTERM before the grace period ended, so the platform escalated to SIGKILL. Or it used more memory than its limit, and the kernel's out-of-memory killer ended it, which Kubernetes reports as OOMKilled. Check the pod's status and the memory graph to tell them apart. By contrast, 143 means it was ended by SIGTERM's default action, and 0 means it shut down cleanly.
StaffA service has a correct SIGTERM handler, but in production every deploy waits the full 30 seconds and the container exits with 137. List the causes and the fixes.
The handler never runs, or runs but never finishes. Check these in order:
- A shell is PID 1. If the container starts the program through
/bin/sh -c, the shell receivesSIGTERMand does not forward it. Use the exec form of the start command so the program is PID 1, or run an init such astinithat forwards signals. - The program is PID 1 and relies on a default action somewhere. The kernel does not apply default actions to PID 1, so any signal without an installed handler is ignored. An init program fixes this too.
- The handler runs but cannot finish.
server.closewaits for every connection to end, and idle keep-alive connections hold it open. SendConnection: closeduring shutdown, or close idle connections explicitly. - There is no internal deadline. A stuck request keeps the process alive until
SIGKILL. Add a deadline shorter than the grace period. - Traffic keeps arriving. If the load balancer is not told to stop routing first, new requests keep the server busy. Fail the readiness check at the start of shutdown.
Scenario Drill
DrillEvery deploy of the photo-upload service drops a handful of uploads, and the logs show nothing wrong. Using this page, explain the likely cause and the fix.
The likely cause is no graceful shutdown. On deploy, the platform sends the old copy SIGTERM. With no handler installed, SIGTERM's default action ends the process instantly. Every upload in progress is cut off halfway, and those users see a failed request. The logs look clean because the process died before it could write anything.
The fix is the handler from section 5:
- Catch
SIGTERMandSIGINT, and make the handler run only once. - Call
server.closeto stop accepting new uploads while in-flight ones finish. - Tell keep-alive clients to close their connections, so the server can finish quickly.
- Close database connections and flush anything buffered.
- Exit with 0, with an internal deadline shorter than the platform's grace period.
Then check the container: the service must actually receive the signal. Start it with the exec form, or put tini in front of it as PID 1. Finally, make sure the grace period is longer than the slowest normal upload. The operating-system fact to state in an interview: deploys do not kill instantly. They ask with SIGTERM first, and your job is to answer that request cleanly before SIGKILL arrives.
Next: a signal carries only a number. Our upload service and its thumbnail program need to exchange real data: which photo to resize, and whether it worked. Page 2.2.4 covers the ways two processes can talk: pipes, sockets, shared memory and message queues, and how to choose between them.