Appearance
2.8.3 — /proc, /sys & /dev
The photo-upload service on a Linux server starts failing with EMFILE: too many open files. You suspect it is leaking file descriptors (2.1.2). How many does it have open, and what are they? No special tool is needed. You can simply list a folder:
sh
ls /proc/4012/fd | wc -l # how many descriptors process 4012 has open
ls -l /proc/4012/fd | head -5 # what each one points toThat folder, /proc/4012/fd, does not exist on any disk. The kernel makes it up the moment you look, from its own records about process 4012. Linux takes the old Unix saying "everything is a file" (2.6.3) further than any other system. Live processes, kernel settings, hardware and devices all appear as files you can read with cat, search with grep and process with any program. This page tours the three places where that happens: /proc, /sys and /dev.
Figure 1 shows the three side by side, and where the contents of each one really come from.
open and read calls. The answers come from the kernel's own records: each process's record for /proc, its list of devices for /sys, and each device's driver for /dev.Read the figure from top to bottom. The tools at the top are ordinary programs, with no special access to the kernel. They open and read files, exactly as they would on a disk. The purple band is the kernel, and the three columns are three folders that exist only inside it. The coloured arrows show where each answer comes from, at the very moment you read it. For /proc, the kernel writes the text from each process's record. For /sys, it reads its list of devices and their settings. For /dev, it hands your read or write to that device's driver. The three sections below take the columns one at a time.
1. /proc: the kernel's records, as files
/proc is a virtual file system: a file system with no disk behind it (2.6.3 showed how the VFS makes this possible). When a program reads a file in /proc, the kernel generates the contents on the spot from its own data. The name stands for process, because that is what it showed first.
One folder per process
For every running process there is a folder named after its PID, such as /proc/4012/. Inside it, the process control block of 2.2.1 is laid out as files. The most useful are:
| File | What it shows |
|---|---|
status | state, threads, memory use, context-switch counts |
cmdline | the command and arguments that started it |
environ | its environment variables |
fd/ | one entry per open file descriptor |
maps | every memory region: code, heap, libraries, stack |
limits | its limits, such as the maximum open files |
cwd | a link to its current directory |
task/ | one folder per thread (2.3.1) |
And /proc/self always means "the process that is reading this", so a program can look at itself. Here the upload service does exactly that.
ts
// self-report.ts — the upload service reads what the kernel knows about it, from /proc
import { readFileSync, readdirSync, readlinkSync, existsSync } from "node:fs";
if (!existsSync("/proc/self/status")) {
console.log("no /proc here: this needs Linux");
} else {
const status = readFileSync("/proc/self/status", "utf8"); // (1)
const pick = (key: string) => status.match(new RegExp(`^${key}:\\s+(.*)$`, "m"))?.[1];
console.log("state:", pick("State"), "| threads:", pick("Threads"), "| resident:", pick("VmRSS"));
console.log("context switches, voluntary / forced:",
pick("voluntary_ctxt_switches"), "/", pick("nonvoluntary_ctxt_switches"));
const fds = readdirSync("/proc/self/fd"); // (2)
console.log(`open descriptors: ${fds.length}`);
for (const fd of fds.slice(0, 4)) console.log(` ${fd} -> ${readlinkSync(`/proc/self/fd/${fd}`)}`); // (3)
const [one, five, fifteen] = readFileSync("/proc/loadavg", "utf8").split(" "); // (4)
console.log(`load average: ${one} (1 min) ${five} (5 min) ${fifteen} (15 min)`);
}
// On Windows, which has no /proc, it prints:
// no /proc here: this needs Linux
//
// On a Linux server it prints something like this (illustrative):
// state: S (sleeping) | threads: 11 | resident: 52340 kB
// context switches, voluntary / forced: 1890 / 41
// open descriptors: 23
// 0 -> /dev/pts/0
// 1 -> /dev/pts/0
// 2 -> /dev/pts/0
// 3 -> anon_inode:[eventpoll]
// load average: 0.42 (1 min) 0.37 (5 min) 0.30 (15 min)(1)statusis plain text, oneName: valuepair per line. The littlepickfunction finds a line by its name and returns the value. Every number here is something earlier pages explained: the state letter from 2.2.1, the thread count, the resident memory from 2.5.4, and the two kinds of context switch.(2)fd/holds one entry per open descriptor, named by its number. Counting them answers the opening question.(3)Each entry is a symbolic link (2.6.1) to what the descriptor really is. In the illustrative output, 0, 1 and 2 are the terminal, a pseudo-terminal from 2.8.2. Descriptor 3 isanon_inode:[eventpoll]: Node's ownepollinstance from 2.7.3, created before your code ran. That is why Node's first file is never descriptor 3 on Linux.(4)/proc/loadavgis system-wide, not per process. Page 2.8.5 explains what load average means.
Two more things make fd/ especially useful. A deleted file that a process still holds open shows up here marked (deleted), which is how you find the "deleted log file still filling the disk" from 2.6.1. And a socket shows as socket:[48211], where the number lets you match it to a network connection.
System-wide files
Other files in /proc describe the whole machine.
| File | What it shows |
|---|---|
/proc/cpuinfo | each CPU core, its model and features |
/proc/meminfo | memory: total, free, available, cached, swap |
/proc/loadavg | load average over 1, 5 and 15 minutes |
/proc/uptime | seconds since boot |
/proc/interrupts | interrupts per device per core (2.7.1) |
/proc/net/tcp | every TCP connection, in hexadecimal |
/proc/meminfo deserves a note. It has both MemFree and MemAvailable. On a healthy server MemFree is often tiny, because spare memory is used as the page cache (2.5.3). MemAvailable estimates how much could be given to programs, including cache the kernel would drop. Watch MemAvailable, not MemFree.
This is how monitoring tools work. ps, top, htop and free have no special access to the kernel. They read these files, add things up, and print a table. When you understand /proc, you understand where every number in those tools comes from.
Kernel settings: /proc/sys
/proc/sys holds the kernel's adjustable settings, one file each, and some of them you can write as well as read. For example:
/proc/sys/vm/swappinesscontrols how eagerly the kernel moves anonymous memory to swap (2.5.3)./proc/sys/fs/file-maxis the maximum number of open files on the whole machine./proc/sys/net/core/somaxconnlimits how many new connections can wait to be accepted by a busy server (2.7.2).
The sysctl command is a friendlier way to read and set them: sysctl vm.swappiness reads the first one. Settings changed this way last until reboot. To keep them, they go in a file under /etc/sysctl.d/.
A warning: /proc can leak secrets
/proc/<pid>/environ holds a process's environment variables, and cmdline holds its arguments. Anyone with access to them can read them, which by default means the same user and the administrator. If the upload service was started with its database password in an environment variable or on its command line, any other program running as the same user can read that password through /proc. That is one reason careful deployments pass secrets in files with tight permissions, or fetch them from a secrets service at startup (8.6.1), and why /proc can be mounted with an option that hides other users' processes.
2. /sys: devices, drivers and kernel objects
/proc grew messy over the years, with files of every kind mixed together. So a second virtual file system, sysfs, mounted at /sys, was added to describe the system's hardware and kernel objects in an orderly tree: every device, the driver it uses, and the settings each one has. Each file usually holds a single value.
A few places you will actually use:
/sys/block/nvme0n1/queue/schedulershows, and lets you change, the disk scheduler for an NVMe disk. It typically reads[none] mq-deadline, the square brackets marking the one in use (2.7.4 explained why NVMe drives usenone)./sys/class/net/eth0/statistics/rx_bytescounts the bytes a network card has received./sys/devices/system/cpu/cpu0/cpufreq/describes a core's clock speed and its power-saving policy./sys/fs/cgroup/is the control-group file system. Every container's limits on CPU and memory are files here, such asmemory.maxandcpu.max, and its current use is in files such asmemory.current. 14.1.3 uses exactly these files to build a container by hand. When a container is killed for using too much memory,memory.eventsin its folder counts each kill.
3. /dev: devices as files
/dev holds device files: special files that stand for hardware, or for something the kernel provides, so that programs can use them with ordinary open, read and write. When a program reads a device file, the kernel passes the request to that device's driver (2.7.1).
There are two kinds, shown by the first letter in ls -l:
- Block devices, marked
b, are storage that can be read in fixed-size blocks at any position:/dev/sdais the first SATA disk,/dev/nvme0n1the first NVMe disk, and/dev/nvme0n1p1its first partition. - Character devices, marked
c, are streams of bytes: terminals such as/dev/ttyand/dev/pts/0, and the special devices below.
Instead of a size, ls -l shows two numbers for each device file, such as 259, 0. These are its major number, which tells the kernel which driver handles it, and its minor number, which tells that driver which particular device. The file is just a name pointing at a driver.
A few special devices are worth knowing by heart.
| Device | Reading gives | Writing does |
|---|---|---|
/dev/null | nothing (end of file at once) | throws the data away |
/dev/zero | endless zero bytes | throws the data away |
/dev/urandom | endless random bytes | adds to the randomness pool |
/dev/tty | the current terminal | writes to the current terminal |
/dev/null is the "bit bucket" of 2.8.1: command > /dev/null discards output. /dev/urandom is a good source of random bytes for keys and tokens (8.2.1). And /dev/shm is not a device but a tmpfs folder in RAM, where the named semaphores of 2.4.2 live.
Who creates all these files? On modern Linux, a background program called udev watches for the kernel's announcements of new hardware, such as a USB stick being plugged in, and creates the matching file in /dev, with the right name and permissions.
4. Putting it together: one investigation
Back to the EMFILE error. Here is how the three file systems answer it, step by step, on the server.
- Find the process.
pgrep -f upload-serviceprints its PID, say 4012. - Count its descriptors.
ls /proc/4012/fd | wc -lsays 1,021. - Check its limit.
grep "open files" /proc/4012/limitssays the limit is 1,024. It is almost out. - See what they are.
ls -l /proc/4012/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | headgroups the targets. Suppose 990 of them aresocket:[...]. - Draw the conclusion. The service is opening network connections, perhaps to cloud storage, and never closing them. The fix is in the code: reuse connections through a pool, and close them on every error path. Raising the limit would only delay the failure.
Every step was plain file reading, with the ordinary tools of 2.8.1, and every number came from a concept you already know.
What the interviewer will push on
"What is /proc, and how do tools like top and ps use it?" They are checking that you know it is a virtual file system generated by the kernel on demand, with one folder per process and system-wide files such as meminfo and loadavg. The tell is saying that ps and top have no special access: they just read and format these files.
"How would you find out what files a process has open?" ls -l /proc/<pid>/fd, or lsof -p <pid>, which reads the same information. Mention that deleted-but-open files appear marked (deleted), and sockets as socket:[inode].
"Free memory on my server is almost zero. Should I worry?" Usually not. Look at MemAvailable in /proc/meminfo, which includes cache the kernel would give up. Worry about swapping and major faults, not MemFree.
"What is the difference between /proc and /sys?" /proc is mainly processes plus a mixture of system information and settings; /sys is an orderly tree of devices, drivers and kernel objects, one value per file, including the cgroup file system that containers use.
"What is /dev/null, and what are major and minor numbers?" A device that discards writes and returns end-of-file on reads. Major and minor numbers on a device file tell the kernel which driver and which device the file stands for.
Something to volunteer that nobody asks for: environment variables are readable through /proc/<pid>/environ by any program running as the same user. That makes "put the secret in an environment variable" weaker than it looks, and is worth mentioning whenever secrets come up.
Recall
/procis a virtual file system generated by the kernel when read./proc/<pid>/shows each process:status,cmdline,environ,fd/,maps,limits,task/;/proc/selfis the reader itself.- System-wide files include
cpuinfo,meminfo(watchMemAvailable),loadavgandinterrupts.ps,topandfreesimply read/proc./proc/sys, viasysctl, holds adjustable kernel settings.environandcmdlinecan leak secrets to the same user. - sysfs at
/sysis an orderly tree of devices and drivers: disk schedulers, network counters, CPU frequency, and/sys/fs/cgroup, where container limits such asmemory.maxlive. /devholds device files: block devices (b, such as/dev/nvme0n1) and character devices (c, such as/dev/tty), each pointing at a driver by its major and minor numbers./dev/null,/dev/zeroand/dev/urandomare special devices;udevcreates device files.- A descriptor leak is found by plain file reading: count
/proc/<pid>/fd, compare withlimits, group the targets.
Self-test: Where does the content of /proc/4012/status come from? Why is Node's first opened file not descriptor 3 on Linux? Why might MemFree be near zero on a healthy server? Where would you look to see a container's memory limit and how often it was killed? What do b, c and the major and minor numbers tell you about a file in /dev?
Quiz Bank
FoundationalWhat is /proc, and how do tools like top and ps use it?
/proc is a virtual file system: no disk is behind it, and the kernel generates each file's contents from its own records at the moment it is read. It has a folder per process, /proc/<pid>/, with files such as status, cmdline, fd/ and maps, and system-wide files such as meminfo, cpuinfo and loadavg. Tools like ps, top, htop and free have no special interface to the kernel: they read these files and format the numbers.
FoundationalWhat are /dev/null, /dev/zero and /dev/urandom?
They are special character device files. /dev/null discards everything written to it and returns end-of-file immediately when read, so it is used to throw output away. /dev/zero returns an endless stream of zero bytes, useful for creating empty files or filling memory. /dev/urandom returns an endless stream of random bytes from the kernel's random number generator, suitable for keys, tokens and other secrets.
AppliedHow would you find out how many files a process has open, and what they are?
On Linux, list /proc/<pid>/fd. Each entry is one open descriptor, so ls /proc/<pid>/fd | wc -l counts them. ls -l /proc/<pid>/fd shows what each points to: a file path, a pipe, a socket written as socket:[inode], or a deleted file marked (deleted). Compare the count with the "Max open files" line in /proc/<pid>/limits. The lsof -p <pid> command shows the same information in more detail.
InterviewWhat is the difference between /proc and /sys?
/proc began as a view of processes and grew to include a mixture of system information and adjustable kernel settings under /proc/sys; its files are often multi-line text. sysfs, mounted at /sys, was added later as an orderly tree of the kernel's devices, drivers and other objects, usually one value per file. Hardware details such as network card counters, disk scheduler settings and CPU frequency live in /sys, as does the cgroup file system at /sys/fs/cgroup, which holds container resource limits and usage.
InterviewA server shows almost no free memory, but the application runs fine. Is there a problem?
Probably not. Linux uses spare memory for the page cache, keeping recently read file data in RAM, so MemFree in /proc/meminfo is often close to zero on a healthy machine. That cache is given up immediately when programs need memory. The meaningful figure is MemAvailable, which estimates how much memory could be given to programs, including reclaimable cache. Real memory trouble shows up as swapping, frequent major page faults, or the out-of-memory killer, not as low MemFree.
StaffYour team stores a database password in an environment variable for the photo-upload service. A security reviewer objects. Using this page, explain the concern and propose a safer design.
A process's environment is readable at /proc/<pid>/environ by any program running as the same user, and by the administrator, and it is inherited by every child process the service starts, including tools that may log their environment when they crash. Command-line arguments are exposed the same way through /proc/<pid>/cmdline, and are often visible to all users through ps. So a secret in the environment is easier to leak than it looks.
There are two safer designs. The service can read the secret at startup from a file readable only by its own user, for example mode 600, or mounted by the platform into memory. Or it can fetch the secret from a secrets manager using the machine's or container's own identity, and keep it only in memory. Run the service as its own dedicated user, so other programs cannot read its /proc entries, and consider mounting /proc with the option that hides other users' processes. Rotate the secret regularly so a leak has a limited lifetime.
Scenario Drill
DrillThe photo-upload service, running in a container, keeps restarting. Using only files under /proc and /sys inside the container, how would you find out whether it is being killed for using too much memory?
Start with the container's cgroup folder under /sys/fs/cgroup/. Read memory.max for the limit and memory.current for the current use. Then read memory.events: its oom_kill count goes up each time the kernel's out-of-memory killer ends a process in this cgroup. If that number rises in step with the restarts, memory is the cause.
To see why memory grows, watch VmRSS in /proc/<pid>/status over time, and look at /proc/<pid>/maps or Node's own heap figures to see which kind of memory is growing. A steady climb under constant load suggests a leak; spikes suggest some requests use a lot of memory at once. The platform will usually confirm it too, with exit code 137 and an OOMKilled status (2.5.3).
Next: one line in the investigation was "check the limit", and one warning was "run the service as its own user". Page 2.8.4 covers who is allowed to do what on a Linux machine: users and groups, the all-powerful root user and how to avoid needing it, and how to run the upload service properly as a systemd service with its logs collected.