Skip to content

Volume ITechnology

2.7.3 — select, poll, epoll & io_uring ​

The photo-upload service, now an event loop (2.7.2), has 10,000 connections open. At any moment perhaps ten of them have data waiting. The other 9,990 are slow phones and idle browsers. Over and over, many thousands of times a second, the loop must ask the kernel one question: which of my connections are ready right now?

How the kernel answers that question decides whether 10,000 connections are easy or impossible. This page follows four generations of answer. select and poll check every connection on every call. epoll remembers which connections you care about and hands back only the ready ones, which is what made the C10k problem easy. io_uring goes a step further and lets a program do most of its input and output with almost no system calls at all.

1. select: "check all of these for me" ​

select arrived with the BSD version of Unix in 1983, and it is still available everywhere. The program builds a set of file descriptors (2.1.2) it wants to watch, one bit per descriptor, and calls select. The call sleeps until at least one of them is ready, then returns, with the set changed to show which ones are ready.

It works. But look at what happens on every single call, with 10,000 connections:

  1. The program builds a set of 10,000 bits, because select changes the set it is given, so it must be rebuilt each time.
  2. The kernel copies the whole set in.
  3. The kernel checks all 10,000 connections, one by one, to see which have data.
  4. The kernel copies the whole set back out.
  5. The program scans all 10,000 bits to find the few that are set.

All of that work is done to find about ten ready connections. The cost of each call grows with the total number of connections, not with the number that are ready. Computer scientists write that as O(n), read "order n": double the connections, double the work per call (4.1 explains this notation). And an event loop makes this call thousands of times a second.

select has a second, harder limit. On most systems the set has a fixed size of 1,024 descriptors, so it cannot watch connection number 1,025 at all.

2. poll: no limit, same scan ​

poll, from the late 1980s, fixed the 1,024 limit. Instead of a bit set, the program passes a list of entries, one per descriptor, each saying "watch this one for reading" and with a space for the kernel's answer. The list can be any length.

But the main cost is unchanged. On every call, the whole list is copied in, every entry is checked, and the program scans the whole list for answers. It is still O(n) per call. Neither call remembers anything from one call to the next. That is the root problem: the program keeps telling the kernel the same 10,000 connections, and the kernel keeps checking all of them.

3. epoll: "remember these, and tell me which are ready" ​

Linux added epoll in 2002. BSD and macOS have an equivalent called kqueue, from 2000, and Windows has I/O completion ports. epoll changes the whole approach. The kernel now remembers the set of connections between calls, and keeps a list of the ones that are ready.

It has three system calls:

  1. epoll_create makes a new, empty interest list inside the kernel and returns a descriptor for it.
  2. epoll_ctl adds a connection to the interest list, removes one, or changes what to watch it for. It is called once per connection, when the connection is opened, not on every loop.
  3. epoll_wait sleeps until at least one watched connection is ready, then returns only the ready ones.

The trick is inside the kernel. When the program adds a connection, the kernel attaches a small hook to it. When data arrives on that connection, the network card's interrupt is handled (2.7.1), the data lands in the connection's buffer, and the hook adds that connection to a ready list. epoll_wait simply hands back what is on the ready list. It never looks at the idle connections at all.

select / poll: every call① copy in all 10,000 descriptors② kernel checks all 10,000③ copy all 10,000 back out④ program scans all 10,000work grows with all connectionsepoll: set up once, then…interest list10,000 connections(added once each)ready listjust the 10 withdata waitingdata arrives → interrupt → added to ready listepoll_wait returns only those 10work grows only with ready connections
Figure 1 — The difference that solved C10k. Left: select and poll copy and check every connection on every call. Right: epoll remembers the connections once; each arrival of data puts that connection on a ready list, and epoll_wait returns only that list.

So the cost of each epoll_wait grows with the number of ready connections, not the total. With 10,000 connections and ten ready, it does work for ten. With 100,000 connections and ten ready, it still does work for ten. Idle connections cost nothing per call. That one change is what turned ten thousand connections from a research problem into an ordinary Tuesday.

The event loop, built from epoll ​

Put the pieces together and you have the core of every event-driven server, including the one inside Node (3.8.1):

  1. Create an epoll instance.
  2. Add the listening socket, the one new connections arrive on.
  3. Loop forever:
    • call epoll_wait, which sleeps, using no CPU, until something is ready;
    • for each ready descriptor, check what it is. If it is the listening socket, accept the new connection and add it to the interest list. Otherwise, read the available data with a non-blocking read, and run the code that handles it;
    • go back to epoll_wait.

No thread per connection, no scanning, no blocking. One thread, sleeping until there is work, then doing exactly the work that is ready. Nginx, Redis, Node through libuv, and most high-performance servers on Linux are this loop.

4. Level-triggered and edge-triggered ​

epoll can report readiness in two different ways, and confusing them causes a nasty bug. Here is a small model of both. A 3,000-byte piece of a photo arrives on one connection, and the handler reads at most 1 KB each time it is woken.

ts
// triggers.ts — a tiny model of epoll's two modes: level-triggered and edge-triggered
type Conn = { name: string; buffered: number; notified: boolean };

function run(mode: "level" | "edge", readUntilEmpty: boolean): void {
  const upload: Conn = { name: "upload-7", buffered: 0, notified: false };
  let received = 0;

  const arrive = (bytes: number) => {                          // (1) the network card delivers data
    upload.buffered += bytes;
    upload.notified = true;                                    //     an "edge": new data appeared
  };
  const isReported = () =>                                      // (2) does epoll_wait report it?
    mode === "level" ? upload.buffered > 0 : upload.notified;

  arrive(3000);                                                // 3,000 bytes of a photo arrive at once
  for (let round = 1; round <= 4; round++) {
    if (!isReported()) { console.log(`  round ${round}: nothing reported, ${upload.buffered} bytes stuck`); continue; }
    upload.notified = false;                                   // (3) reporting consumes the edge
    do {
      const n = Math.min(1024, upload.buffered);               // (4) read() takes at most 1 KB
      upload.buffered -= n; received += n;
    } while (readUntilEmpty && upload.buffered > 0);           // (5) keep reading until EAGAIN?
    console.log(`  round ${round}: read so far ${received}, still buffered ${upload.buffered}`);
  }
}

console.log("level-triggered, one read per wake-up:"); run("level", false);
console.log("edge-triggered, one read per wake-up:");  run("edge", false);
console.log("edge-triggered, read until empty:");      run("edge", true);

// Prints:
// level-triggered, one read per wake-up:
//   round 1: read so far 1024, still buffered 1976
//   round 2: read so far 2048, still buffered 952
//   round 3: read so far 3000, still buffered 0
//   round 4: nothing reported, 0 bytes stuck
// edge-triggered, one read per wake-up:
//   round 1: read so far 1024, still buffered 1976
//   round 2: nothing reported, 1976 bytes stuck
//   round 3: nothing reported, 1976 bytes stuck
//   round 4: nothing reported, 1976 bytes stuck
// edge-triggered, read until empty:
//   round 1: read so far 3000, still buffered 0
//   round 2: nothing reported, 0 bytes stuck
//   round 3: nothing reported, 0 bytes stuck
//   round 4: nothing reported, 0 bytes stuck
  • (1) When data arrives, it is added to the connection's buffer in the kernel, and the connection is flagged as having had new data. Engineers call that moment of change an edge, a word borrowed from electronics.
  • (2) This line is the whole difference. Level-triggered mode reports a connection whenever its buffer is not empty. Edge-triggered mode reports it only when new data has arrived since the last report.
  • (3) Each report uses up the edge.
  • (4) The handler reads at most 1 KB per read call.
  • (5) The fix for edge-triggered mode: keep calling read until the buffer is empty, which on a real non-blocking socket means until read returns EAGAIN.

Now read the output.

  • Level-triggered keeps reporting the connection, round after round, as long as data remains. It is forgiving: a handler that reads a little each time still gets everything eventually. That is select's and poll's behaviour, and epoll's default.
  • Edge-triggered with one read per wake-up is the bug. It reported once, the handler read 1 KB, and 1,976 bytes were left stuck in the buffer forever. No new data arrived, so there was no new edge, so epoll never reported the connection again. That upload would simply hang.
  • Edge-triggered, reading until empty, gets all 3,000 bytes in the first round, and is correct.

Why use edge-triggered mode at all? Because it reports each connection once per arrival instead of on every call while data remains, which saves work when there are many busy connections, and because it lets several threads share one epoll instance without all being woken for the same connection. The price is the rule: in edge-triggered mode, always drain the connection until EAGAIN.

5. io_uring: almost no system calls at all ​

epoll solved the scanning problem. One cost remains. It is readiness-based (2.7.2): epoll_wait says a connection is ready, and then the program makes a separate read system call to get the data, and a write call to reply. Every operation still costs at least one crossing into the kernel, and 2.1.2 showed each crossing has a fixed price, raised further by the security fixes of 2018.

io_uring, added to Linux in 2019 by Jens Axboe, removes most of those crossings. It is completion-based. The program and the kernel share two ring buffers, circular queues in memory that both can see, so neither has to make a system call to put something in or take something out.

  1. The submission queue. The program writes requests into it: "read from connection 17 into this buffer", "write this reply to connection 42", "read 64 KB of this file". It can write hundreds before telling the kernel anything.
  2. The completion queue. The kernel writes results into it: "the read on connection 17 finished, 812 bytes", "the write to connection 42 finished".
upload serviceuser modekerneldoes the worksubmission ringread 17 · write 42 · read file …completion ring17: 812 bytes · 42: done …both rings live in memory shared by both sides
Figure 2 — io_uring's two rings. The program writes requests into the submission ring and reads results from the completion ring. The kernel does the opposite. Because both rings are shared memory, many operations need no system call at all.

When the program does need to wake the kernel, one io_uring_enter system call can submit hundreds of requests and collect hundreds of results. In some modes, a kernel thread keeps watching the submission ring, so the program never makes a system call at all. It is the same ring-of-buffers idea that network cards use with DMA (2.7.1), now offered to ordinary programs.

io_uring has another strength. It works for ordinary files, which epoll cannot help with, because a disk file always counts as "ready" even when reading it will wait for the disk (2.7.2). With io_uring, file reads become truly asynchronous, with no background thread pool needed.

It also has a cost worth knowing. Its large, fast-moving code has been a common source of kernel security bugs. In 2023, Google reported that most of the kernel exploits submitted to its bug-bounty programme the year before had targeted io_uring, and disabled it on Android, ChromeOS and its production servers. Some container platforms block it by default too. It is powerful and still maturing.

InterfaceYearCost per waitRemembers?Files?
select1983all descriptors, max 1,024nono
polllate 1980sall descriptorsnono
epoll / kqueue2002 / 2000ready descriptors onlyyesno
io_uring2019batched, few system callsyesyes

What the interviewer will push on ​

"Why does select scale poorly and epoll scale well?" They are checking whether you know the reason, not just the fact. select and poll remember nothing between calls, so every call copies in and checks every descriptor: O(n) in all connections. epoll keeps an interest list in the kernel and a ready list filled as data arrives, so epoll_wait costs only the ready connections. Mention select's 1,024 limit.

"How does epoll know a connection is ready without checking it?" When a connection is added, the kernel attaches a hook to it. When data arrives and the interrupt is handled, the hook puts the connection on the ready list. The tell is connecting this back to interrupts.

"What is the difference between level-triggered and edge-triggered?" Level-triggered reports a descriptor whenever data is waiting; edge-triggered only when new data arrives. In edge-triggered mode you must read until EAGAIN, or leftover data is stranded. Describing that bug from experience is a strong signal.

"What is io_uring, and how is it different from epoll?" Completion-based I/O through two shared ring buffers, so many operations need no system call, and files can be read asynchronously. epoll only reports readiness, after which each read and write is still a system call.

"What does Node use underneath?" libuv, which uses epoll on Linux, kqueue on macOS and I/O completion ports on Windows, plus a thread pool for files.

Something to volunteer that nobody asks for: the move from select to epoll is a move from asking to being told, the same move as from polling to interrupts in 2.7.1, one level up. Seeing that pattern repeat shows you understand why each step was faster.

Recall

  • An event loop must ask "which connections are ready?" thousands of times a second. select (1983, max 1,024 descriptors) and poll (no limit) copy in and check every descriptor on every call: O(n), with nothing remembered.
  • epoll (Linux, 2002), like kqueue and Windows completion ports, keeps an interest list in the kernel, set once per connection with epoll_ctl. Arriving data, via its interrupt, puts a connection on a ready list, and epoll_wait returns only those. Idle connections cost nothing per call.
  • The event loop: epoll_wait, handle each ready descriptor (accept new connections or read data), repeat. Nginx, Redis and Node via libuv work this way.
  • Level-triggered mode reports while data is waiting; edge-triggered only on new arrivals, so the handler must read until EAGAIN or data is stranded.
  • io_uring (Linux, 2019) is completion-based: requests and results go through shared ring buffers, batching hundreds of operations into few or no system calls, and it works for files. It has been a frequent source of security bugs.

Self-test: Count the work select does on one call with 10,000 connections and ten ready. What puts a connection on epoll's ready list? Why did the edge-triggered model leave 1,976 bytes stuck? What does io_uring remove that epoll still requires? Why can epoll not help with reading ordinary files?

Quiz Bank ​

FoundationalWhy does select scale poorly and epoll scale well?

select and poll remember nothing between calls. Every call passes the whole set of descriptors, the kernel checks every one to see which are ready, and the program scans the whole result. That is O(n) work per call in the total number of descriptors, however few are actually ready, and select also stops at about 1,024 descriptors.

epoll remembers. Each descriptor is registered once with epoll_ctl. The kernel keeps a ready list and adds a descriptor to it when data arrives. epoll_wait returns only the ready ones, so idle connections cost nothing per call, which makes tens of thousands of connections practical.

FoundationalWhat are the three epoll system calls and what does each do?

epoll_create makes a new epoll instance, with an empty interest list inside the kernel, and returns a descriptor for it. epoll_ctl adds a descriptor to the interest list, removes one, or changes which events to watch for; it is called once per connection rather than on every loop. epoll_wait sleeps until at least one watched descriptor is ready, or a timeout passes, and returns only the ready descriptors and what each is ready for.

AppliedWhat is the difference between level-triggered and edge-triggered epoll, and what bug does edge-triggered mode invite?

In level-triggered mode, epoll_wait reports a descriptor every time it is called while the descriptor has data waiting. In edge-triggered mode, it reports the descriptor only when new data has arrived since the last report.

The bug: with edge-triggered mode, a handler that reads only part of the available data, for example one read of 1 KB when 3 KB arrived, will not be told again unless more data arrives, so the rest sits in the buffer and the connection appears to hang. The rule is to make descriptors non-blocking and, on each report, keep reading until read returns EAGAIN.

InterviewWhat is io_uring and why is it faster than epoll for some workloads?

io_uring is a Linux interface, added in 2019, in which the program and the kernel share two ring buffers: a submission queue where the program places requests such as reads and writes, and a completion queue where the kernel places results. Because the rings are shared memory, placing requests and collecting results needs no system call, and when one is needed, a single call can submit and complete hundreds of operations. In some modes a kernel thread polls the submission ring, so no system calls are needed at all.

It is faster than epoll for workloads with very many small operations, because epoll only reports readiness and every read and write is still its own system call. It also supports truly asynchronous file I/O, which epoll cannot provide.

InterviewDescribe the event loop at the heart of a server like Nginx or Node.

The server puts its sockets in non-blocking mode and creates an epoll instance, then registers the listening socket. It then loops forever:

  1. It calls epoll_wait, which sleeps without using CPU until some descriptors are ready.
  2. For each ready descriptor, it either accepts new connections and registers each with epoll, or reads the available data and runs the handler code for it, writing replies without blocking.
  3. It calls epoll_wait again. Nothing ever blocks, and the loop only does work for connections that actually have something to do, so one thread, or one per core, can serve tens of thousands of connections.
StaffYour team proposes moving a high-throughput storage service from epoll plus a thread pool to io_uring. What benefits and risks would you lay out?

Benefits. File and network operations both become asynchronous through one interface, so the thread pool used for blocking file calls can go away. Many operations can be submitted and completed per system call, or with none in polling mode, reducing the per-operation crossing cost that dominates workloads of many small requests. Fewer context switches and less locking between pool threads follow.

Risks. io_uring has had a long series of serious security bugs and is disabled or blocked in some environments, including some container platforms and Google's own fleet, so it may not be available where the service runs. Its API is more complex, and bugs around buffer lifetimes are easy to make, because the kernel uses your buffers after the call returns. Behaviour and features vary with kernel version.

A sensible plan is to measure the real share of time spent in system calls first, prototype on the target kernel, keep the epoll path as a fallback, and check the platform's security policy before committing.

Scenario Drill ​

DrillThe photo-upload service, rewritten in C for speed using edge-triggered epoll, occasionally leaves uploads hanging forever partway through, though the clients are still connected and sending. Using this page, find the bug.

This is the edge-triggered stranding bug. In edge-triggered mode, epoll_wait reports a connection only when new data arrives. If the handler, on each report, calls read once with a fixed buffer, say 4 KB, and the kernel had 10 KB waiting, 6 KB is left in the socket's buffer. If the client then pauses, for example waiting for an acknowledgement before sending more, no new data arrives, so no new edge, so epoll never reports that connection again. The upload hangs, with data sitting unread in the kernel.

The fix: make every socket non-blocking and, on each report, loop on read until it returns EAGAIN, processing each piece. Alternatively switch to level-triggered mode, which keeps reporting the connection while data remains, at some extra cost per call. When a server must also treat connections fairly, it can read up to a budget per connection per round and re-queue the connection itself, so one fast client cannot starve the others.

Next: we now know how a server waits for input efficiently. The last question is how much input and output a machine can actually do. Page 2.7.4 explains the three numbers that describe storage and network performance, IOPS, throughput and latency, how they connect through a simple law about queues, and two techniques that every fast server uses: zero-copy and backpressure.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Volume I — Technology
Part 0 — Retention & Notation
Part 1 — Foundations of Computation
Part 2 — Operating Systems
2.1 The Kernel
2.2 Processes
2.3 Threads & Scheduling
2.4 Concurrency Under the Hood
2.5 Memory Management
2.6 File Systems
2.7 Input/Output
2.8 Linux for Engineers
2.9 Windows & macOS in Practice
Part 3 — Languages, Compilers & Runtimes
3.6 JavaScript & V8
3.7 TypeScript
3.8 Node.js Internals
3.13 Python & CPython
3.14 FastAPI & Production Python
Part 4 — Data Structures & Algorithms
4.0 Classic Problems — Before NeetCode
4.4 Arrays & Hashing — Problems
4.5 Two Pointers
4.6 Sliding Window
4.8 Stack — Problems
4.9 Linked List — Problems
4.11 Binary Search — Problems
4.13 Trees
4.14 Tree — Problems
4.15 Trie — Problems
4.17 Heap — Problems
4.18 Backtracking — Problems
4.19 Graphs
4.20 Graph — Problems
4.21 Advanced Graph — Problems
4.23 1-D DP — Problems
4.24 2-D DP — Problems
4.26 Greedy — Problems
4.27 Interval — Problems
4.28 Math & Geometry — Problems
4.30 Bit Manipulation — Problems
Part 5 — Networking & the Web
5.3 IP
5.4 Transport
5.6 HTTP
Part 6 — Frontend Engineering
6.1 Browser Internals
6.2 HTML & CSS Mastery
6.3 JavaScript in the Browser
6.4 React, from First Component to Full Shop
6.5 Frameworks & Rendering
6.8 UI Engineering & Patterns
Part 7 — Databases & Storage
7.2 SQL Mastery
7.3 Storage Engine Internals
7.4 Transactions
7.5 NoSQL Taxonomy
7.8 Analytics & Big Data
Part 8 — Security, Identity & Cryptography
8.2 Cryptography
8.3 Protocols of Trust
8.4 Identity: AuthN & AuthZ
8.5 Application Security
8.6 Infrastructure Security
Part 9 — Software Design (LLD)
9.2 OOP Foundations
9.3 Design Principles
9.4 Design Patterns
9.5 Concurrency
9.6 API Design
9.7 LLD Interview Machines
9.9 Node/Express Production Playbook
Part 10 — Distributed Systems & HLD
10.7 Consistency & Consensus
10.8 Messaging & Microservices
10.14 Caching
Building blocks from the design seat
Part 11 — System Design Case Studies
Part 12 — AI & ML Engineering
12.1 What AI Engineering Is
12.2 The Math You Need
12.3 Classical Machine Learning
12.4 Data for ML
12.5 Deep Learning
12.6 Embeddings & Vector Search
12.7 Transformers & LLMs
12.8 Building with LLMs
12.9 Retrieval-Augmented Generation
12.10 Agents
12.11 Adapting Models
12.12 Production AI
12.13 Multimodal AI
12.14 AI Security & Governance
12.15 Case Studies & Capstone
Part 13 — Cloud, DevOps & Infrastructure
13.11 Git, from First Commit to Team Workflow
Part 14 — Containers: Docker & Kubernetes
14.1 Foundations
14.2 Docker
14.3 Kubernetes
14.4 Kubernetes on Azure
Part 15 — The Craft & Career
Nothing picked yet