Appearance
2.7.2 — Blocking, Non-Blocking & the C10k Problem
Fifty people press "upload" at the same moment. For each photo, the upload service has to wait about 200 milliseconds for its storage to respond. Here are two versions of the service, both running on one thread. The only difference is how that thread waits.
ts
// blocking-vs-nonblocking.ts — 50 uploads arrive together; each needs 200 ms of waiting on storage
import { createServer, request, type Server } from "node:http";
const sleepBlocking = (ms: number) => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
function start(blocking: boolean): Promise<Server> {
const server = createServer((req, res) => {
if (blocking) {
sleepBlocking(200); // (1) the thread itself waits
res.end("saved");
} else {
setTimeout(() => res.end("saved"), 200); // (2) ask to be told; keep serving others
}
});
return new Promise((ok) => server.listen(0, () => ok(server)));
}
async function fire(server: Server, n: number): Promise<number> {
const port = (server.address() as { port: number }).port;
const t = performance.now();
await Promise.all(Array.from({ length: n }, () => new Promise<void>((done) => {
request({ port, path: "/upload", agent: false }, (res) => { res.resume(); res.on("end", done); }).end();
})));
return performance.now() - t;
}
for (const blocking of [true, false]) {
const server = await start(blocking);
const ms = await fire(server, 50);
console.log(`${blocking ? "blocking " : "non-blocking"}: 50 uploads finished in ${(ms / 1000).toFixed(2)} s`);
server.close();
}
// Prints:
// blocking : 50 uploads finished in 10.45 s
// non-blocking: 50 uploads finished in 0.25 s(1)In the first version, the thread itself waits 200 ms for each upload. While it waits, it can do nothing else. So the 50 uploads are served one after another: 50 × 0.2 s = 10 seconds. The last person waits ten seconds for a 200 ms job.(2)In the second version, the thread says "tell me in 200 ms" and immediately goes back to accept the next upload. All 50 waits overlap. Every upload finishes about 200 ms after it arrived, and the whole batch is done in a quarter of a second.
Same work, same single thread, forty times faster. The difference is between blocking and non-blocking waiting. It is one of the most important decisions in building a server, and this page explains both kinds, the two server designs they lead to, and the wall the simpler design hits at about ten thousand connections.
1. Blocking I/O: the thread sleeps until the data arrives
Blocking I/O is the default for system calls such as read. If the upload service calls read on a network connection and no data has arrived yet, the system call does not return. The kernel puts the thread to sleep, in the blocked state from 2.2.1. When the data arrives, the network card raises an interrupt (2.7.1), the kernel copies the data into the program's buffer, wakes the thread, and read finally returns with the bytes.
This is wonderfully simple to program. The code reads from top to bottom, as if input and output were instant: read the request, read the photo, save the photo, send the reply. A sleeping thread also uses no CPU.
But it has a hard limit. While a thread is blocked, it can do nothing else. To serve a second connection at the same time, you need a second thread. To serve a thousand at once, you need a thousand threads.
2. Non-blocking I/O: ask, and get an answer at once
A program can open a connection in non-blocking mode instead. Then read always returns immediately. If data is there, it returns the data. If nothing has arrived yet, it returns at once with a special error code, EAGAIN, which means "nothing yet, try again later". The thread is never put to sleep. It stays in control, and can go and do something useful for another connection.
That gives one thread a superpower: it can juggle many connections, because none of them can trap it. But it creates a new question at once. If read only says "nothing yet", when should the program try again? Trying every connection over and over is polling, the waste from 2.7.1. The program needs the kernel to tell it which connections have data ready. That tool is I/O multiplexing, and the next page, 2.7.3, is entirely about it.
3. Two ways to build a server
This choice produces two completely different server designs.
Thread per connection. Every new connection gets its own thread, which uses blocking calls. The code is simple and linear. Much of the web was built this way. The Apache web server's classic modes gave each connection its own process or thread, and many Java servers gave each request a thread.
The event loop. One thread, or one per CPU core, watches all the connections at once. It asks the kernel "which of these connections are ready?", does a small piece of work for each ready one, never blocks, and goes back to ask again. The code is harder to write, because a request's work is split into many small steps that run whenever their data is ready. Nginx, Redis and Node.js are all built this way (3.8.1).
Our demo showed the event loop's advantage with only one thread on each side. A thread-per-connection server would have handled the 50 uploads quickly too, using 50 threads. The interesting question is what happens with 10,000.
4. The C10k problem
In 1999, a programmer named Dan Kegel wrote a web page asking how to serve ten thousand simultaneous connections on one machine. It became known as the C10k problem, "C" for clients. Computers were fast enough. The software design was not.
Here is where thread per connection struggles at that scale.
- Memory. Each thread needs a stack. The default reserves 8 MB of address space, and each thread actually touches some tens of kilobytes or more of real memory for its stack and the kernel's records (2.3.1). Ten thousand threads use hundreds of megabytes just to exist.
- Scheduling. When traffic arrives, thousands of threads wake, run briefly and sleep again. Every wake-up is a context switch, with its direct cost and its cold caches (2.2.1). The CPU spends more and more of its time switching instead of serving.
- Most of them are doing nothing. A typical connection to a busy site is idle most of the time: a browser holding a connection open, a mobile app waiting for the user, a slow phone network. Ten thousand threads, mostly asleep, still cost memory and scheduler work, to wait.
The event loop removes all three. One thread per core, no matter how many connections, so no per-connection stack. No context switch per request, because the same thread handles everything. And an idle connection costs only the kernel's small record of it, a few kilobytes of buffers, not a whole thread.
Nginx was written in 2004 specifically to solve the C10k problem, and it does. Today a single well-tuned machine can hold hundreds of thousands, even millions, of mostly idle connections, which people now call the C10M problem, "ten million".
The event loop's weakness
The event loop has one rule that must never be broken: no step may block, and no step may take long. If one step runs a slow calculation, or calls a blocking function, the one thread is stuck, and every connection waits. Our demo's blocking version is exactly that mistake: one blocked wait stalled all 50 uploads.
This is why Node insists on "don't block the event loop". CPU-heavy work, such as resizing photos, belongs on worker threads (2.3.1) or in separate processes (2.2.2). The event loop is perfect for work that is mostly waiting, and bad for work that is mostly computing.
5. The five ways to wait for input
W. Richard Stevens, in his classic book on Unix network programming, sorted every way a program can wait for input into five models. They differ in two questions: who waits for the data to arrive, and who waits while it is copied from the kernel into the program's buffer.
| Model | Waiting for data | Copying into the program |
|---|---|---|
| 1. Blocking | thread sleeps in read | thread sleeps in read |
| 2. Non-blocking | thread keeps asking, gets EAGAIN | thread waits in read |
| 3. Multiplexing | thread sleeps in select/epoll, for many connections | thread waits in read |
| 4. Signal-driven | kernel sends a signal when ready | thread waits in read |
| 5. Asynchronous | nothing; thread carries on | kernel copies, then reports "done" |
Walk down the table.
- Blocking is the default from section 1. Simple, one thread per connection.
- Non-blocking on its own means asking again and again, which is polling. It is rarely used alone.
- Multiplexing combines non-blocking connections with one call that sleeps until any of many connections is ready. This is what event loops use, and the next page explains
select,pollandepoll. - Signal-driven I/O has the kernel send a signal when data is ready. Signals carry almost no information (2.2.3), so this model is awkward and rarely used.
- Asynchronous I/O is different in kind. The program says "read this into that buffer, and tell me when you are done", and gets on with other work. The kernel does both the waiting and the copying, and then reports completion. Linux's
io_uringand Windows's I/O completion ports work this way.
The first four are called readiness-based: the kernel says "you can read now", and the program then makes the read call itself. The fifth is completion-based: the kernel says "your read has finished, the data is in your buffer". That distinction matters for the next page.
Where Node fits
Node uses model 3 for network connections, through a library called libuv, which uses epoll on Linux, kqueue on macOS and I/O completion ports on Windows (3.8.2). For ordinary files, most operating systems historically offered no good readiness model, because a disk file is always "ready" even when reading it will block on the disk. So libuv does file work with blocking calls on a small pool of background threads, and reports back to the event loop when they finish. Your code sees one consistent, non-blocking API for both.
What the interviewer will push on
"What is the difference between blocking and non-blocking I/O?" They are checking whether you can say what happens to the thread. Blocking: the call does not return until data is there, so the thread sleeps and can do nothing else. Non-blocking: the call returns at once, with data or with EAGAIN. The strong answer adds the consequence for architecture: blocking leads to thread per connection, non-blocking plus multiplexing leads to an event loop.
"What is the C10k problem, and how was it solved?" Serving ten thousand connections at once. Thread per connection fails on memory per thread and on context switches among thousands of mostly idle threads. The solution was non-blocking sockets with scalable multiplexing, epoll or kqueue, in an event loop, as in Nginx, Redis and Node.
"Why is an event loop great for a chat server and bad for video encoding?" Chat is mostly waiting, so one thread can serve thousands of idle connections. Encoding is mostly computing, so a single loop thread is stuck on each video and every other connection waits. Put CPU work on worker threads or processes.
"What is the difference between readiness-based and completion-based I/O?" Readiness: the kernel says a connection can be read, and the program then calls read. Completion: the program asks for the read up front, the kernel does it, and reports when the data is already in the buffer.
"How does Node do file I/O without blocking?" libuv runs file operations on a small thread pool with blocking calls and reports completion to the event loop, because files have no useful readiness signal on most systems.
Something to volunteer that nobody asks for: the event loop's win is not that it is faster per request. It is that an idle connection costs almost nothing. Most connections on real servers are idle most of the time, and that is exactly the case thread per connection handles worst.
Recall
- Blocking I/O puts the thread to sleep until data arrives: simple top-to-bottom code, but one thread per connection. Non-blocking calls return at once, with data or
EAGAIN, so one thread can juggle many connections. - That choice leads to two designs: thread per connection (classic Apache, many Java servers) or an event loop (Nginx, Redis, Node). In the demo, one thread served 50 overlapping 200 ms uploads in 0.25 s non-blocking and 10.45 s blocking.
- The C10k problem (Dan Kegel, 1999): ten thousand mostly idle threads cost memory and context switches. Event loops cost almost nothing per idle connection.
- The event loop must never block or compute for long; CPU-heavy work goes to worker threads or processes.
- Stevens's five models: blocking, non-blocking, I/O multiplexing, signal-driven and asynchronous. The first four are readiness-based; asynchronous (
io_uring, Windows completion ports) is completion-based. Node uses multiplexing for sockets and a thread pool for files.
Self-test: Why did the blocking version take 10 seconds for 50 uploads? What does EAGAIN mean, and what should a program do when it gets it? Name the three costs that sink thread per connection at ten thousand connections. Why is an event loop a poor fit for resizing photos? What is the difference between being told "ready" and being told "done"?
Quiz Bank
FoundationalWhat is the difference between blocking and non-blocking I/O, and what architecture does each imply?
With blocking I/O, a call such as read does not return until data is available, so the thread sleeps in the meantime and can do nothing else. That implies thread per connection: each simultaneous connection needs its own thread. It is simple to write, but costs memory and context switches at scale.
With non-blocking I/O, the call returns immediately, either with data or with EAGAIN, meaning nothing is ready yet. Combined with a multiplexing call that reports which connections are ready, it implies an event loop: one thread serving many connections by doing a little work for whichever are ready. It is harder to write and scales to very large numbers of connections.
FoundationalWhat is the C10k problem and how was it solved?
The C10k problem, named in 1999, is serving ten thousand simultaneous connections on one machine. With thread per connection, that needs about ten thousand threads, each with a stack and kernel records, and when traffic arrives the scheduler switches among thousands of them, wasting time and cache. Most of those threads are just waiting on idle connections. The solution is non-blocking sockets plus scalable I/O multiplexing such as epoll on Linux or kqueue on BSD and macOS, used by an event loop with about one thread per core. Nginx, Redis and Node.js work this way.
AppliedWhy is the single-threaded event loop great for I/O-bound servers but bad for CPU-bound work?
For I/O-bound work, such as thousands of connections mostly waiting on the network or storage, one event-loop thread can serve them all: while any connection waits, the thread works on others, and an idle connection costs almost nothing. For CPU-bound work, the same single thread is a weakness: any long computation blocks the whole loop, so every other connection stalls until it finishes. That is why runtimes like Node move CPU-heavy work, such as image processing, to worker threads or separate processes, keeping the loop free for I/O.
InterviewExplain readiness-based versus completion-based I/O.
In readiness-based I/O, the program asks the kernel which connections are ready, using select, poll or epoll, and the kernel answers "these can be read now". The program then makes its own read call for each, which copies the data. Every operation still costs at least one system call.
In completion-based I/O, the program submits the whole operation up front, "read into this buffer", and continues. The kernel waits for the data, copies it, and reports that the operation has completed. Linux's io_uring and Windows's I/O completion ports work this way. Completion-based I/O can batch many operations with few system calls, and works for files, which have no useful readiness signal.
InterviewA Node service's response times jump for every request whenever one particular endpoint is called. What is the likely cause?
That endpoint is almost certainly blocking the event loop: doing heavy synchronous computation, or calling a synchronous API such as a *Sync file function or a CPU-heavy library call. Node runs all JavaScript on one thread, so while that endpoint's code runs, no other request can be processed, and every request in flight waits for it.
Confirm it by measuring event-loop delay, or by profiling during the slowdown. Fix it by making the work asynchronous, moving CPU-heavy work to a worker thread pool or a separate service, or splitting it into small chunks that yield back to the loop between them.
StaffYou must choose between a thread-per-connection design and an event loop for a service that holds 50,000 long-lived, mostly idle connections and occasionally does 50 ms of CPU work per message. Decide and justify.
The connection pattern decides the core design: 50,000 mostly idle connections favour an event loop, where an idle connection costs only kernel buffers and a small record, instead of a thread with its stack and scheduler cost. Thread per connection would mean 50,000 threads, gigabytes of stack in total once touched, and heavy context switching whenever traffic bursts.
The occasional 50 ms of CPU work is the risk: done on the loop, it would stall every connection on that thread for 50 ms at a time. So pair the loop with a bounded worker pool, about one thread per core, and hand CPU work to it through a queue, returning results to the loop. Run one event loop per core, or several processes, to use the whole machine. Alternatively, a runtime with very cheap threads, such as Go's goroutines or Java's virtual threads, gives thread-per-connection code with event-loop costs, if the team prefers that style.
Scenario Drill
DrillThe photo-upload service handles each upload on its own thread in a Java-style framework. At 3,000 simultaneous uploads from slow mobile connections, the server runs out of memory and response times climb, although CPU use is low. Using this page, explain and propose a fix.
Each upload holds one thread for its whole duration, and a slow mobile connection can take tens of seconds to send a photo. So 3,000 slow uploads mean 3,000 threads, nearly all blocked waiting for the next few kilobytes. Each thread costs stack memory and kernel records, which is why memory runs out, and every time data arrives the scheduler must wake and switch between thousands of threads, which is why response times climb while the CPU does little real work. This is the C10k problem at a smaller number.
The fix is to stop paying a thread per waiting connection. Use non-blocking I/O with an event loop, or a framework built on one, so that slow connections cost only their buffers. Stream each upload to storage as it arrives instead of holding the whole photo in memory. If the team wants to keep the thread-per-request style, a runtime with lightweight threads, such as Java's virtual threads, gives the same benefit with familiar code. Either way, put a limit on simultaneous uploads so a surge cannot exhaust memory.
Next: an event loop needs the kernel to answer one question very quickly: "which of my 10,000 connections have data ready right now?". Page 2.7.3 follows the four generations of answers, from select, which checked every connection every time, to epoll, which made C10k easy, to io_uring, which removes most of the system calls altogether.