Appearance
2.7.4 — I/O Performance: IOPS, Throughput, Latency & Backpressure
The photo-upload service needs a new cloud disk. The cloud provider offers two at the same price. One advertises "up to 1,000 MB/s". The other advertises "up to 16,000 IOPS". Which one should the service buy?
The answer depends entirely on what the service does with the disk, and getting it wrong makes a service slow no matter how much it costs. This page covers:
- the three numbers that describe any storage device or network link;
- the simple law about queues that ties them together;
- why most code cannot reach a disk's advertised speed;
- sending data without copying it;
- making a fast sender slow down for a slow receiver;
- and the order in which a hard disk should serve its requests.
1. Three numbers: IOPS, throughput and latency
Every input and output device, whether a disk, an SSD or a network link, is described by three numbers.
- IOPS, input/output operations per second: how many separate requests the device can complete each second, however small each one is.
- Throughput, also called bandwidth: how many bytes it can move each second, usually in MB/s or GB/s.
- Latency: how long one request takes, from asking to getting the answer.
The first two are linked by the size of each request:
\text{throughput} = \text{IOPS} \times \text{request size}
Read that as "bytes per second equals operations per second times bytes per operation". A disk doing 10,000 requests a second of 4 KB each moves about 40 MB a second. The same disk doing 1,000 requests a second of 1 MB each moves about 1,000 MB a second.
So the two cloud disks are really advertising two different limits. Which limit you hit depends on your request size.
| Device (rough figures) | Random 4 KB IOPS | Sequential throughput |
|---|---|---|
| Spinning hard disk | 100–200 | 150–250 MB/s |
| SATA SSD | tens of thousands | about 550 MB/s |
| NVMe SSD (PCIe 4.0) | hundreds of thousands | several GB/s |
The hard disk row is the clearest. Its throughput is respectable, because once the head is in place it reads a continuous stream quickly. But its IOPS is tiny, because every random request needs a seek of several milliseconds (2.6.3). At 150 requests a second of 4 KB each, a hard disk doing random reads moves only 150 \times 4\text{ KB} = 600 KB a second, hundreds of times below its sequential speed.
Which number matters for which workload
- Many small, scattered requests: a database looking up rows, a service checking thousands of tiny files, the photo index being updated. The limit is IOPS. Buy for IOPS and low latency.
- Few large, continuous requests: streaming a video, backing up a disk, copying a large file. The limit is throughput. Buy for MB/s.
For the photo-upload service, saving one 5 MB photo is a large write, but updating the index and creating thumbnails are many small ones. If it also serves thumbnails to browsers, it does thousands of small reads. It probably needs IOPS more than raw throughput.
Cloud providers sell exactly these two limits, separately. For example, one common general-purpose cloud disk type comes with a baseline of 3,000 IOPS and 125 MB/s, and you pay more to raise either one independently (figures at the time of writing; providers change them). A disk that hits its IOPS limit does not fail. Requests simply queue, and latency climbs.
Latency, and why averages lie
Latency is the number users feel. An NVMe SSD read takes tens of microseconds; a hard-disk read takes milliseconds; a request to a disk across a network adds the network's round trip. But averages hide the worst cases. A disk with an average latency of 1 ms might take 50 ms for one request in a hundred, while its internal garbage collection runs (2.6.3). That slow hundredth is called the 99th percentile, written p99, and it is usually what decides whether a service feels fast (10.9 returns to this).
2. Little's law: why you need many requests in flight
Here is a puzzle. An NVMe SSD advertises 500,000 IOPS. The upload service reads from it one request at a time, waiting for each read before starting the next. Each read takes 100 microseconds. How many IOPS does the service get?
One request every 100 microseconds is 1 \div 0.0001 = 10{,}000 requests a second. The service gets 10,000 IOPS from a device that can do 500,000. The device is not slow. The service is not asking it enough at once.
The rule behind this is Little's law, proved by John Little in 1961. It holds for any queue in a steady state, from a supermarket to a disk:
L = \lambda \times W
Read it as "the number in the system equals the arrival rate times the time each one spends". Here:
- L is the number of requests in flight at once. For storage this is called the queue depth.
- \lambda, the Greek letter lambda, is the rate: requests completed per second, the IOPS.
- W is the time each request takes: the latency.
Turn it round to see what you need: \lambda = L \div W. With a latency of 100 microseconds:
| Requests in flight (L) | IOPS you get (L \div W) |
|---|---|
| 1 | 10,000 |
| 8 | 80,000 |
| 32 | 320,000 |
| 64 | the device's limit, about 500,000 |
To reach a fast device's advertised IOPS, a program must keep many requests outstanding at once. That is exactly why NVMe has thousands of deep queues (2.7.1), why io_uring lets a program submit hundreds of requests in one go (2.7.3), and why databases issue reads in parallel.
There is a direct, practical consequence for Node. Node performs file operations on libuv's thread pool, which has 4 threads by default (3.8.2). At most 4 file operations can be in flight. With 100-microsecond reads, Little's law caps the whole service at 4 \div 0.0001 = 40{,}000 file operations a second, whatever disk it runs on. Raising the pool size with the UV_THREADPOOL_SIZE setting raises that ceiling.
Little's law works the other way too, for diagnosing slowness. If a disk is already at its limit and requests keep arriving, L grows: requests pile up in the queue, and since W = L \div \lambda, each one waits longer. When a device is saturated, extra load turns into extra latency, not extra throughput. That is the signature of an IOPS-limited service: the IOPS graph is flat at the limit, and the latency graph shoots up.
3. Measuring honestly: the page cache gets in the way
A warning before you benchmark a disk. The kernel's page cache keeps recently used file data in RAM (2.6.2). If you read the same file twice, the second read comes from RAM and tells you nothing about the disk. Writes return before reaching the disk at all. Many "my SSD does 5 GB/s!" results are really measuring memory.
Real storage benchmarks either use files far larger than RAM, or ask the kernel to bypass the cache with a flag called O_DIRECT, and they control the request size and the queue depth explicitly. On Linux, the standard tool for this is fio. When you read a cloud provider's IOPS figure, it was measured at a particular request size and queue depth. Your workload may differ on both.
4. Zero-copy: the fastest copy is the one you skip
Consider the upload service sending a stored photo to a browser, the simple way:
readcopies the photo's bytes from the page cache into the program's buffer. That is a copy from the kernel to the program.writecopies them from the program's buffer into the socket's buffer in the kernel. That is a copy from the program back to the kernel.- The network card then takes them from the socket buffer by DMA (2.7.1).
The program never looked at the bytes. It just carried them out of the kernel and back in again: two copies and two crossings of the user–kernel wall (2.1.2), for nothing.
Zero-copy techniques remove that detour. The best known is the system call sendfile, which tells the kernel "send this file, from this position, to that socket, yourself". The data goes from the page cache to the network card without ever entering the program. On modern hardware, the network card can often take the data straight from the page cache's pages, so it is not copied at all inside the kernel either.
That is how web servers such as Nginx serve static files at the full speed of the network, and how the Kafka message broker sends stored messages to many readers cheaply (10.8.2). Memory-mapped files (2.5.4) are another way of avoiding a copy, for programs that do need to look at the data.
5. Backpressure: making a fast sender wait for a slow receiver
The last idea is about speed mismatches. In the upload service, a photo arrives from a fast network connection and is written to slower storage. What happens if data arrives faster than it can be written?
ts
// backpressure.ts — a fast reader feeding a slow writer, with and without listening to "slow down"
import { Writable } from "node:stream";
import { once } from "node:events";
function slowStorage(): Writable {
return new Writable({
highWaterMark: 64 * 1024, // (1) "my buffer is full at 64 KB"
write(chunk, _enc, done) { setTimeout(done, 1); }, // (2) takes 1 ms per chunk
});
}
async function upload(respectBackpressure: boolean): Promise<void> {
const storage = slowStorage();
const chunk = Buffer.alloc(16 * 1024); // the network hands us 16 KB at a time
let peak = 0;
for (let i = 0; i < 2000; i++) { // a 32 MB photo, arriving fast
const ok = storage.write(chunk); // (3) false means "please wait"
peak = Math.max(peak, storage.writableLength);
if (!ok && respectBackpressure) await once(storage, "drain"); // (4) wait until it catches up
}
storage.end();
await once(storage, "finish");
console.log(`${respectBackpressure ? "with backpressure " : "ignoring backpressure"}: ` +
`most held in memory at once = ${(peak / 2 ** 20).toFixed(2)} MB`);
}
await upload(false);
await upload(true);
// Prints:
// ignoring backpressure: most held in memory at once = 31.25 MB
// with backpressure : most held in memory at once = 0.06 MB(1)The slow storage says how much it is willing to hold in its buffer before it wants the sender to wait: 64 KB.(2)It takes 1 ms per chunk, far slower than the loop can produce chunks.(3)writealways accepts the chunk, but its return value matters.falsemeans "my buffer is over the limit; please stop until I tell you".(4)When it saysfalse, the careful version waits for thedrainevent, which means "I have caught up, carry on".
Ignoring the signal, the service ended up holding almost the whole 32 MB photo in memory at once. Listening to it, it never held more than 64 KB. Now multiply by a hundred simultaneous uploads: 3 GB of memory against 6 MB. The first version runs fine in testing and runs out of memory under real load.
That feedback, the slow side telling the fast side to wait, is called backpressure. It turns up at every level of a system:
- Pipes have it built in: a full pipe puts the writer to sleep (2.2.4).
- TCP has it built in: a receiver advertises how much buffer space it has left, and the sender slows to match (Part 5).
- Node streams have it if you use them properly.
stream.pipelineconnects streams and handlesdrainfor you (3.8.4). - Message queues need it designed in, as bounded queues that reject or slow producers when full (10.8.1).
The alternative, an ever-growing buffer, only moves the problem. "Just buffer it" turns a speed mismatch into a memory leak that ends with the out-of-memory killer (2.5.3). Any time a fast stage feeds a slow stage, either the slow stage pushes back, or memory grows without limit.
6. Disk scheduling: the elevator algorithm
One more performance idea comes straight from the physics of spinning disks. When many requests are waiting for a hard disk, the order in which the kernel sends them decides how far the head has to travel, and head movement is the slow part (2.6.3).
Here is the classic textbook example. A disk has tracks numbered 0 to 199. The head is at track 53. Requests are waiting for tracks 98, 183, 37, 122, 14, 124, 65 and 67, in that order of arrival. Count how many tracks the head crosses in total for three different orders.
| Policy | Order served | Tracks crossed |
|---|---|---|
| First come, first served | 98, 183, 37, 122, 14, 124, 65, 67 | 640 |
| Shortest seek first | 65, 67, 37, 14, 98, 122, 124, 183 | 236 |
| Elevator (sweep up, then down) | 65, 67, 98, 122, 124, 183, 37, 14 | 299 |
- First come, first served swings the head wildly back and forth: 53 to 98, all the way out to 183, back to 37, and so on. 640 tracks.
- Shortest seek first always goes to the nearest waiting track. It travels only 236 tracks. But it has the starvation problem from 2.3.2: if new requests keep arriving near the head, a request far away may wait forever.
- The elevator algorithm, also called SCAN, moves the head in one direction, serving every request it passes, and only reverses when there is nothing further ahead. It works exactly like a lift in a building, which is where the name comes from. Here it sweeps up from 53 to 183, then back down to 37 and 14: 299 tracks. That is a little more than shortest-seek-first, but every request is guaranteed to be reached within one sweep, so nothing starves. The variant shown, which reverses at the last request instead of at the very edge of the disk, is called LOOK.
The elevator machine in 9.7.28 uses the same algorithm for the same reason: a physical thing that is slow to reverse, serving requests spread along a line.
Linux still offers disk schedulers built on these ideas, such as mq-deadline, which sorts requests to reduce seeking but also gives each one a deadline so none waits too long. For NVMe SSDs, which have no head and many parallel queues (2.7.1), the usual choice is none: reordering requests buys nothing when every position costs the same to reach.
What the interviewer will push on
"What is the difference between IOPS and throughput, and which matters for a database?" They are checking that you can connect them: throughput equals IOPS times request size. A database does many small random reads and writes, so it is limited by IOPS and latency. A video server reading large files in order is limited by throughput.
"An SSD advertises 500,000 IOPS but our service gets 10,000. Why?" Little's law. With one request in flight and 100 µs latency, the most you can get is 10,000 a second. You need many requests outstanding, a deep queue, to reach the device's limit. Bonus: Node's default four-thread pool caps file operations the same way.
"What happens to latency when a disk hits its IOPS limit?" Requests queue, the number in flight grows, and by Little's law each waits longer. Throughput stays flat at the limit and latency climbs. That is how you recognise saturation on a dashboard.
"What is zero-copy, and where is it used?" Moving data from a file to a socket inside the kernel, with sendfile, instead of copying it into the program and back. Web servers use it for static files, and Kafka uses it to serve messages.
"What is backpressure, and what happens without it?" A slow consumer signalling a fast producer to wait. Without it, buffers grow until memory runs out. In Node, respect write() returning false and wait for drain, or use pipeline.
Something to volunteer that nobody asks for: most disk benchmarks that look too good to be true are measuring the page cache. Real numbers come from files larger than RAM or from O_DIRECT, at the request size and queue depth the workload actually uses.
Recall
- Three numbers describe any I/O device: IOPS (requests per second), throughput (bytes per second) and latency (time per request). Throughput = IOPS × request size; small random work is IOPS-bound, large sequential work is throughput-bound.
- A hard disk does only 100–200 random IOPS but streams 150+ MB/s; SSDs do tens to hundreds of thousands of IOPS. Watch tail latency such as p99, not just averages.
- Little's law: in flight = rate × time, L = \lambda W. With 100 µs latency, one request at a time gives only 10,000 IOPS; reaching a fast device's limit needs a deep queue depth. Node's 4-thread pool caps file operations this way. At saturation, extra load becomes latency.
- Benchmarks must defeat the page cache (huge files or
O_DIRECT) and state request size and queue depth. - Zero-copy (
sendfile) sends file data to a socket without passing it through the program. Backpressure makes a fast producer wait for a slow consumer; ignoring it held 31 MB in the demo against 64 KB. - On hard disks, request order decides head travel: first come, first served crossed 640 tracks, shortest seek first 236 but can starve, and the elevator algorithm 299 with no starvation. NVMe SSDs need no reordering.
Self-test: Compute the random-read throughput of a hard disk doing 150 IOPS of 4 KB. Why did one-at-a-time reads get 10,000 IOPS from a 500,000-IOPS device? What queue depth reaches 320,000 IOPS at 100 µs? What two copies does sendfile avoid? What happens to memory when a producer ignores write() returning false?
Quiz Bank
FoundationalWhat is the difference between IOPS, throughput and latency?
IOPS is how many separate input or output operations a device completes per second, regardless of size. Throughput is how many bytes it moves per second. Latency is how long one operation takes. They are linked: throughput equals IOPS times the size of each operation. Workloads of many small random operations, such as databases, are limited by IOPS and latency; workloads of few large sequential operations, such as video streaming or backups, are limited by throughput.
FoundationalWhat is backpressure, and why does it matter?
Backpressure is a signal from a slower consumer to a faster producer to wait until the consumer catches up. It matters because otherwise the data the consumer cannot yet handle has to be buffered somewhere, and the buffer grows without limit until memory runs out. Pipes and TCP provide it automatically; Node streams provide it when you respect write() returning false and wait for drain, or use pipeline; queues between services need bounded sizes to provide it.
AppliedUse Little's law to explain why a program reading one block at a time cannot reach an SSD's advertised IOPS.
Little's law says the number of requests in flight equals the rate multiplied by the time each takes: L = \lambda W, so \lambda = L / W. A program that waits for each read before starting the next has L = 1. With an SSD latency W of 100 microseconds, the rate is at most 1 / 0.0001 = 10{,}000 reads a second. To reach, say, 320,000 IOPS at the same latency, about 32 requests must be in flight at once. That is why fast storage needs deep queues, parallel requests, or interfaces like io_uring that submit many requests together.
InterviewYou are choosing a cloud disk for a database doing mostly small random reads and writes. One option offers high throughput, another high IOPS. Which matters, and what would change for a video-streaming server?
For the database, IOPS and latency matter. Each query touches small records scattered across the disk, so the limit is how many operations per second the disk can do, not bytes per second. A disk with huge throughput but low IOPS will run out of operations; requests will queue and latency will climb. Choose for high IOPS and low latency, which also means SSD rather than hard disk.
A video-streaming server reads large chunks of each file in order: few operations, each big. There, raw throughput is the limit and IOPS hardly matters, so choose for MB/s, and use sendfile and a CDN to deliver the bytes efficiently. Characterise the workload as small-random or large-sequential before choosing; the two lead to opposite decisions.
InterviewWhat is zero-copy, and why does it make static file serving fast?
Serving a file the simple way reads it into the program's buffer, copying it out of the kernel's page cache, and writes it to the socket, copying it back into the kernel, though the program never inspects it. Zero-copy interfaces such as sendfile ask the kernel to move the data from the file to the socket directly, so it never crosses into the program, avoiding both copies and both crossings of the user–kernel boundary. On modern hardware the network card can often read the page cache's pages directly. Web servers like Nginx use it for static files, and Kafka uses it to serve stored messages.
StaffA service's dashboard shows disk IOPS flat at exactly 3,000 for an hour while request latency rises from 5 ms to 400 ms. Explain what is happening and the options.
A flat line at a round number is a provisioned limit: the cloud disk allows 3,000 IOPS and the service is asking for more. By Little's law, when the rate is capped and requests keep arriving, the number waiting grows, and the time each spends grows with it, so latency climbs while throughput stays flat. Nothing is broken; the disk is saturated.
The options:
- Buy more IOPS, or a faster disk type. This is the quickest fix.
- Do fewer I/O operations: cache more in memory, batch small writes into larger ones, or fix queries that read more rows than needed.
- Spread the load over several disks or replicas.
- Move the busiest data to a faster storage tier. Also check whether the page cache or the database's own cache is undersized, since a few gigabytes more RAM can remove most reads.
Scenario Drill
DrillThe photo-upload service streams each upload straight to cloud storage. Under a load test, memory climbs steadily until the process is killed, although each upload is only a few megabytes. Using this page, find the bug and fix it.
The service reads from the incoming network connection faster than cloud storage accepts writes, and it ignores backpressure. Each chunk read from the client is passed to the storage stream's write, which returns false once its buffer is full, but the code keeps writing anyway. So the unsent part of every upload piles up in memory. With many uploads at once and slow storage, memory grows until the out-of-memory killer ends the process.
Fix it by letting the slow side set the pace. Connect the incoming request stream to the storage stream with stream.pipeline, which pauses reading from the client whenever the storage stream's buffer is full and resumes on drain. Pausing the read in turn fills the socket's buffer, and TCP's own flow control tells the client to slow down. Then memory per upload stays at a fixed small amount, and the load test's memory stays flat. Also put a limit on simultaneous uploads, so a surge queues at the door instead of inside the process.
Next: Part 2 has built the operating system from the inside: the kernel, processes, threads, locks, memory, files and I/O. 2.8.1 turns from concepts to the tool you will use to see all of it on a real Linux machine: the shell, its three streams, and the pipes that connect small programs into powerful ones.