Appearance
2.8.5 — Observing a Running System
It is 3 a.m. and the photo-upload service is slow. Uploads that took 200 ms now take four seconds. You have a terminal on the server and no idea why.
The temptation is to guess: restart it, add memory, blame the network. Experienced engineers do something else. They look, in a fixed order, at what each resource on the machine is doing, and let the numbers point to the cause. Every part of the operating system you have met in this Part, the CPU and scheduler, memory, disks, the network, processes and the kernel, can be inspected from that terminal with a handful of tools. This page is a field guide to those tools, what each number actually means, and a method for finding the bottleneck instead of guessing. It ends with the two kernel features that turn ordinary processes into containers.
1. Start with a method, not a tool
The most useful method is short enough to remember at 3 a.m. Brendan Gregg, a performance engineer, called it the USE method. For every resource, meaning each CPU, memory, each disk, each network link, check three things:
- Utilisation: how busy is it? "The disk is busy 95 % of the time."
- Saturation: is work queueing for it? "Twelve requests are waiting for the disk."
- Errors: is it failing? "The network card is dropping packets."
Saturation is the one people forget, and often the most telling. A resource can be 100 % busy and fine, if nothing is waiting. Once work queues for it, every request waits longer, which is Little's law from 2.7.4: at saturation, extra load turns into extra latency.
The table below says where to find each number. The rest of the page explains the tools.
| Resource | Utilisation | Saturation | Errors |
|---|---|---|---|
| CPU | top: us + sy | load average, vmstat r | rare |
| Memory | free: available | vmstat si/so, major faults | OOM kills in dmesg |
| Disk | iostat -x: %util | iostat -x: queue, await | dmesg I/O errors |
| Network | sar -n DEV: throughput | dropped packets, ss queues | ip -s link errors |
Figure 1 draws the same map as a picture: where on the machine each tool looks.
When you investigate, read the figure from the bottom up. Start with the four resources, asking the three USE questions of each, because the bottleneck is almost always one of them. Then check the kernel's log, which reports failures no application will mention. Only when you know which resource is struggling do you move to the top row and look inside the one process that is using it. The rest of this page follows that order.
2. A first look: uptime, top and the load average
text
$ uptime
03:12:44 up 41 days, 2:07, 1 user, load average: 7.82, 3.10, 1.20The load average is the average number of threads that are running, ready to run, or waiting in uninterruptible sleep, over the last 1, 5 and 15 minutes. The three numbers show the trend. Here, 7.82 now against 1.20 a quarter of an hour ago means the trouble started recently and is getting worse.
To judge it, compare with the number of CPU cores. On a 4-core server, a load of 4 means the cores are fully used with nothing waiting; 7.82 means roughly four threads are always waiting for a core.
But one detail of Linux matters a great deal. Linux counts threads in the uninterruptible D state (2.2.1) as part of the load, and those are usually waiting for a disk, not for the CPU. So a high load average does not always mean the CPU is busy. It can mean "many threads are stuck waiting for storage". The next tool tells the two apart.
top, and its friendlier cousin htop, show a live table of processes, sorted by CPU use. The summary line at the top splits CPU time into parts:
text
%Cpu(s): 12.0 us, 6.1 sy, 0.0 ni, 40.2 id, 41.0 wa, 0.0 hi, 0.7 si, 0.0 st| Field | Time spent… |
|---|---|
us | running user-mode code: your programs |
sy | in the kernel: system calls (2.1.2) |
ni | running programs with a raised nice value (2.3.3) |
id | idle |
wa | idle, but waiting for disk I/O to finish |
hi / si | handling interrupts and their bottom halves (2.7.1) |
st | stolen: waiting for the hypervisor, on a virtual machine |
In that example, wa is 41 %. The CPU is not the bottleneck at all: it is mostly waiting for the disk. Combined with the load average, the picture is "threads piling up in disk waits".
The st field matters in the cloud. A virtual machine shares a physical machine with others (14.1.2). If st is high, the hypervisor is giving your machine's CPU time to someone else, and the fix is a bigger or dedicated machine, not your code.
3. vmstat: the whole machine, once a second
vmstat 1 prints one line per second covering CPU, memory, swap and I/O. It is often the single most useful command, because it shows trends instead of one moment.
text
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 9 0 312140 40120 6021448 0 0 61240 812 4120 9810 12 6 40 41 0
1 11 0 310872 40120 6021804 0 0 63008 604 4377 9932 11 6 39 43 0The columns that matter most:
r: threads ready to run, waiting for a CPU. Consistently more than the number of cores means CPU saturation.b: threads blocked in uninterruptible sleep, usually on disk. Here 9 to 11: many threads are stuck on storage.si,so: memory swapped in and out per second. Anything steadily above zero suggests thrashing (2.5.3). Here they are zero, so memory is not the problem.bi,bo: blocks read from and written to disk per second.biaround 61,000 means heavy reading.cs: context switches per second.us sy id wa st: the same CPU split astop.
Reading the two lines together tells a story: memory is fine, the CPU is mostly idle, many threads are blocked, the disk is reading flat out, and the CPU is waiting on I/O. Something is reading the disk very hard.
4. Disk and network: iostat, ss and friends
iostat -xz 1 shows each disk separately, with the IOPS, throughput and latency of 2.7.4.
text
Device r/s rkB/s w/s wkB/s r_await aqu-sz %util
nvme0n1 3001.0 60150.0 52.0 812.0 38.10 114.5 100.0r/sandw/sare reads and writes per second: IOPS. Here reads are pinned at about 3,000 a second.rkB/sandwkB/sare throughput.r_awaitis the average time a read takes, including time spent queueing: 38 ms, for a device that should answer in well under 1 ms.aqu-szis the average number of requests in the queue: 114. That is saturation.%utilis how much of the time the device was busy. For an SSD with many parallel queues, 100 % does not necessarily mean full, which is why queue size and wait time are the better guides.
A flat line of exactly 3,000 reads a second, a long queue and a high wait time is the signature from 2.7.4's Staff question: the disk has hit a provisioned IOPS limit. The question becomes: why is the service suddenly reading so much?
For the network, the common tools are:
ss -tanplists every TCP connection with its state and owning process.ss -ssummarises them. Thousands of connections in one state is a clue. A growingRecv-Qon a connection means the program is not reading data that has arrived. A growingSend-Qmeans the other side, or the network, cannot keep up.sar -n DEV 1shows throughput per network card each second;ip -s linkshows each card's error and drop counts.lsof -ilists open network connections by process.lsof -p <pid>lists everything a process has open, the same information as/proc/<pid>/fd(2.8.3).
5. Inside one process: pidstat, strace and perf
Once you know which resource is struggling, find which process is using it. pidstat -d 1 shows disk reads and writes per process, and pidstat -u 1 shows CPU per process. In our story, pidstat -d points straight at the upload service, reading about 60 MB a second.
To see what it is doing, two tools look inside a single running process.
strace: the system calls
strace shows every system call a process makes (2.1.2). It attaches to a running process with -p, and -f follows its threads.
sh
sudo strace -f -p 4012 -e trace=openat,read -tt 2>&1 | head
sudo strace -f -c -p 4012 # after Ctrl+C, a table of call counts and total timeThe -c form is a quick profile: it might show that the service made 180,000 read calls of 4 KB each in ten seconds, on files under /var/lib/photos/thumbs. Now the story is nearly complete: the service is re-reading thumbnails from disk on every request.
strace is invaluable, but it slows the traced program a great deal, because the kernel stops the program at every system call. Use it briefly in production, and with care.
perf: where the CPU time goes
When the problem is CPU, perf samples where each core is spending its time, many times a second, with very little overhead. sudo perf top shows the busiest functions live. perf record and perf report save samples to study later, and they can be drawn as a flame graph, where the widest bars are where the time goes (15.5 shows how to read one). For Node programs, Node's own profiler, node --prof or --cpu-prof, gives JavaScript-level detail.
The process's own view
Programs can measure themselves too. For a Node service, the most important number is how long its single event loop is being held up (2.7.2). Node can measure that directly.
ts
// loop-delay.ts — measure how long the event loop is stuck, before and during a blocking job
import { monitorEventLoopDelay } from "node:perf_hooks";
import { createHash } from "node:crypto";
const ms = (ns: number) => (ns / 1e6).toFixed(1) + " ms";
async function measure(label: string, work: () => void): Promise<void> {
const h = monitorEventLoopDelay({ resolution: 10 }); // (1) sample every 10 ms
h.enable();
const end = Date.now() + 2000;
while (Date.now() < end) { // run for 2 seconds
work();
await new Promise((r) => setTimeout(r, 5)); // (2) give the loop a turn
}
h.disable();
console.log(`${label}: median ${ms(h.percentile(50))}, p99 ${ms(h.percentile(99))}, max ${ms(h.max)}`);
}
await measure("idle service ", () => {});
await measure("resizing on the loop ", () => { // (3) ~100 ms of CPU per call
let s = "x";
for (let i = 0; i < 60_000; i++) s = createHash("sha256").update(s).digest("hex");
});
// Printed on a Windows laptop:
// idle service : median 15.7 ms, p99 16.7 ms, max 24.1 ms
// resizing on the loop : median 108.3 ms, p99 186.8 ms, max 186.8 ms(1)monitorEventLoopDelaysets a repeating timer, here every 10 ms, and records how late each one fires. A timer can only fire when the loop is free, so lateness measures how long the loop was busy.(2)Each round yields to the loop briefly, as a real server does between requests.(3)In the second run, each round does about 100 ms of CPU work on the loop, standing in for resizing a photo in the wrong place.
The idle figures include the 10 ms sampling interval itself, and Windows timers fire in steps of about 15 ms, which is why an idle loop reads about 16 ms here. What matters is the jump: once CPU work runs on the loop, the median delay rises to over 100 ms and the worst to nearly 190 ms. Every request arriving during that time waits that long before it is even looked at. Exporting this one number to a dashboard catches the most common Node performance bug before users report it.
6. The kernel's own messages: dmesg and the journal
The kernel writes important events to its own log, which dmesg prints, and which journalctl -k shows with timestamps. This is where you find things no application log will mention:
Out of memory: Killed process 4012 (node): the OOM killer (2.5.3).task node:4012 blocked for more than 120 seconds: a thread stuck in theDstate (2.4.3).I/O error, dev nvme0n1, sector 91442: a failing disk.TCP: request_sock_TCP: Possible SYN flooding: more new connections than the server can accept.
journalctl -u photo-upload --since "30 min ago" shows the service's own output for the same period (2.8.4), so you can line up "when did the kernel complain" with "what was the service doing".
Back to 3 a.m.
Figure 2 shows the whole investigation as a funnel. Each tool narrowed the search, and each step ruled something out.
Put the evidence together. The load average was rising, but the CPU was mostly idle and waiting for I/O. Memory and swap were fine. The disk was pinned at its 3,000 IOPS limit with a long queue. pidstat pointed at the upload service. strace -c showed it re-reading thumbnails from disk on every request. A release that afternoon had removed the in-memory thumbnail cache. The fix was to restore the cache, not to restart, add memory or blame the network. Every step was a measurement, and each one ruled something out.
7. Where containers come from: namespaces and cgroups
One last pair of kernel features belongs in any tour of a running Linux system, because they explain what a container actually is. Every container you will ever run is an ordinary Linux process, which you can see with ps and inspect with every tool on this page, wrapped in two things.
- Namespaces control what a process can see. Each kind of namespace gives a process its own private view of one part of the system. It can have its own list of processes, in which it is PID 1. It can have its own tree of files, its own network interfaces and ports, and its own hostname. Inside, it looks like a separate machine.
- cgroups, control groups, control what a process can use. They cap a group's CPU, memory and I/O, and count what it uses. You met their files under
/sys/fs/cgroupin 2.8.3, and the CPU throttling they cause in 2.3.3.
There is no "container" object in the Linux kernel at all. A tool such as Docker sets up the namespaces and cgroups, provides a file system image, and then starts your program with fork and exec (2.2.2). That is why a container starts in milliseconds and why all containers on a machine share its one kernel. Part 14 builds this properly: 14.1.3 explains each namespace and cgroup, and 14.1.4 builds a working container by hand from exactly these pieces. Virtual machines, the heavier alternative that gives each guest its own kernel, are in 14.1.2.
What the interviewer will push on
"The service is slow. Walk me through how you would investigate." They are checking for a method, not a list of tools. A good answer has three steps. First, check utilisation, saturation and errors for each resource, using the load average, vmstat, iostat, free and dmesg. Second, narrow down to a process with pidstat or top. Third, look inside it with strace, perf or the runtime's own metrics. The tell is ruling resources out with evidence before acting.
"What does the load average mean?" The average number of threads running, ready to run, or in uninterruptible sleep, over 1, 5 and 15 minutes; compare with the core count. The strong answer adds that Linux includes D-state threads, so a high load can mean disk waits rather than CPU.
"top shows high wa. What does that tell you?" The CPU is idle but waiting for disk I/O, so look at the disks with iostat, not at CPU-heavy code. High st on a virtual machine means the hypervisor is taking your CPU time.
"What is the difference between strace and perf?" strace shows system calls, one by one, and slows the program heavily. perf samples where CPU time goes, with low overhead, and is suited to CPU problems.
"What is a container, in terms of the kernel?" An ordinary process isolated by namespaces, which limit what it sees, and limited by cgroups, which cap what it uses, sharing the host's kernel.
Something to volunteer that nobody asks for: saturation is often more telling than utilisation. A disk at 100 % busy with an empty queue is fine; a disk at 60 % with a queue of 100 is the bottleneck. Asking "what is waiting?" before "what is busy?" finds problems faster.
Recall
- The USE method: for every resource check utilisation, saturation and errors, before guessing. Saturation means work is queueing, and by Little's law it turns into latency.
- The load average counts threads running, ready or in uninterruptible sleep, over 1, 5 and 15 minutes; on Linux a high load can mean disk waits.
topsplits CPU intous,sy,wa(waiting on disk),siandst(stolen by the hypervisor). vmstat 1showsr(ready threads),b(blocked),si/so(swapping) and I/O each second.iostat -xshows each disk's IOPS, wait time and queue size.ss,sar -n DEVandlsofcover the network.pidstatfinds the process; strace shows its system calls (slows it down); perf samples CPU time cheaply. Node'smonitorEventLoopDelaymeasures how long the event loop is stuck.dmesgshows OOM kills, hung tasks and disk errors.- A container is an ordinary process with namespaces (what it can see) and cgroups (what it can use), sharing the host kernel; Part 14 builds one.
Self-test: A 4-core server shows a load average of 8 with 40 % wa. What is likely happening? Which vmstat columns would show thrashing? What pattern in iostat reveals a provisioned IOPS limit? Why should strace be used only briefly in production? What does an event-loop delay of 180 ms mean for users?
Quiz Bank
FoundationalWhat does the load average mean, and how do you judge whether it is high?
The load average is the average number of threads that are running, ready to run, or in uninterruptible sleep, over the last 1, 5 and 15 minutes. Compare it with the number of CPU cores: a load equal to the core count means fully busy with nothing waiting, and well above it means work is waiting. On Linux, threads waiting uninterruptibly for disk are included, so a high load with idle CPUs and high I/O wait points to storage rather than to CPU. Comparing the three numbers shows whether things are getting better or worse.
FoundationalWhat does strace do, and when is it invaluable?
strace prints every system call a program makes, with its arguments, result and errors, either from the start or by attaching to a running process with -p. It is invaluable when a program fails without a useful message: it shows the exact file it tried to open and the ENOENT it got, the network address it is connecting to, or the call it is stuck in. With -c it summarises call counts and time, which quickly shows a program making far too many small calls. It slows the traced program heavily, so use it briefly in production.
AppliedExplain the USE method and apply it to a disk.
The USE method says: for every resource, check utilisation (how busy it is), saturation (how much work is queued for it) and errors (how often it fails). For a disk, all three come from familiar places. Utilisation is the share of time it was busy, with its IOPS and throughput, from iostat -x. Saturation is its average queue size and request wait time, also from iostat -x. Errors are I/O error messages in dmesg and the disk's own health counters. A disk with a long queue and a rising wait time is saturated and is causing latency, even if its utilisation figure does not look extreme.
InterviewA service is slow. How would you find out whether the cause is CPU, memory, disk or network?
Work through the resources and rule them out with evidence. Start with uptime for the load-average trend and dmesg for kernel errors such as OOM kills or disk errors. Then vmstat 1: r well above the core count suggests CPU saturation; non-zero si/so suggests swapping; a high b and high wa suggest disk. iostat -xz 1 confirms disk saturation through queue size and wait time. free -m shows available memory. For the network, sar -n DEV 1, ss -s and interface error counts. Once a resource is identified, pidstat or top finds the process, and strace, perf or the runtime's metrics show what it is doing.
Interviewtop shows the CPU mostly idle, but the load average is high. How is that possible on Linux?
Linux's load average counts not only threads that are running or waiting for a CPU, but also threads in uninterruptible sleep, the D state, which are usually waiting for disk or network storage. If many threads are stuck waiting on slow I/O, they raise the load while using no CPU, so top shows a mostly idle CPU, typically with a high wa. The next step is to look at the disks with iostat -x and at which processes are blocked, rather than at CPU usage.
StaffDesign the minimum set of operating-system and runtime signals you would put on a dashboard for the photo-upload service, and say what each catches.
For each resource, one utilisation and one saturation signal, plus the errors that matter:
- CPU: utilisation per core, and run-queue length or load average relative to cores; plus stolen time if it runs on virtual machines. Catches CPU saturation and noisy neighbours.
- Memory: available memory, swap activity and major page faults, plus OOM-kill counts from the kernel or the cgroup's
memory.events. Catches leaks and thrashing before the kill. - Disk: IOPS and throughput against provisioned limits, request latency and queue size. Catches hitting an IOPS cap.
- Network: throughput, retransmissions and dropped packets, and connection counts by state. Catches saturation and connection leaks.
- Process: open descriptors against the limit, thread count, restarts and exit codes. Catches
EMFILEbefore it happens and crash loops. - Runtime: event-loop delay percentiles, and heap size against resident memory. Catches blocked loops and leaks.
Pair them with the service's own request rate, error rate and latency percentiles, so every OS symptom can be matched against user impact.
Scenario Drill
DrillAfter a deploy, the photo-upload service's p99 latency doubles, but CPU, memory, disk and network on the host all look normal. Using this page, where would you look next?
When the machine's resources look healthy, look inside the process. For a Node service, the first suspect is the event loop: check event-loop delay with monitorEventLoopDelay or the service's metrics. A new piece of synchronous work, such as a large JSON parse or an image operation added in the deploy, can hold the loop for tens of milliseconds, which barely shows in host CPU on a many-core machine but delays every request queued behind it.
Next, check whether the process is waiting on something outside the host: ss -tanp for connections to a dependency piling up, and strace -c -p briefly, to see whether it is spending time in connect, poll or read on a slow downstream service. Check the container's cgroup for CPU throttling (2.3.3), which causes latency spikes without high average CPU. Finally, compare the deploy's changes against what the measurements point to. The method is the same: measure, rule out, narrow down.
Next: Part 2 has taught the operating system through Linux, because that is what servers and containers run. But your own working day is probably on Windows or macOS. 2.9.1 opens the Windows and macOS kernels, and shows that under their very different surfaces sit the same processes, threads, memory and files you now understand.