Appearance
2.6.3 — Real File Systems & the VFS
The photo-upload service stores 20,000 thumbnails, each 1 KB. Together that is 20 MB. Here is the service reading them two ways: as 20,000 separate files, and as the same 20 MB stored in one file.
ts
// many-small-files.ts — the same 20 MB as 20,000 thumbnails, or as one file
import { mkdirSync, writeFileSync, readdirSync, statSync, readFileSync, rmSync } from "node:fs";
import { join } from "node:path";
rmSync("thumbs", { recursive: true, force: true });
mkdirSync("thumbs");
const thumb = Buffer.alloc(1024, 7);
for (let i = 0; i < 20_000; i++) writeFileSync(join("thumbs", `t-${i}.jpg`), thumb); // (1)
writeFileSync("all-thumbs.bin", Buffer.alloc(20_000 * 1024, 7)); // (2)
let t = performance.now();
let total = 0;
for (const name of readdirSync("thumbs")) total += statSync(join("thumbs", name)).size; // (3)
console.log(`size of folder by stat-ing 20,000 files: ${(performance.now() - t).toFixed(0)} ms`);
t = performance.now();
for (const name of readdirSync("thumbs")) readFileSync(join("thumbs", name)); // (4)
console.log(`read 20,000 small files: ${(performance.now() - t).toFixed(0)} ms`);
t = performance.now();
readFileSync("all-thumbs.bin"); // (5)
console.log(`read one 20 MB file: ${(performance.now() - t).toFixed(0)} ms (same ${total / 2 ** 20} MB)`);
// Printed on a Windows laptop with an SSD (a second run gave 1,640 / 16,092 / 13 ms):
// size of folder by stat-ing 20,000 files: 1358 ms
// read 20,000 small files: 18293 ms
// read one 20 MB file: 28 ms (same 19.53125 MB)(1)Write 20,000 tiny files, each 1 KB.(2)Write the same total amount of data as one file.(3)Work out the folder's total size the way most programs do: list the folder, then ask for each file's details withstat.(4)Read every small file.(5)Read the one big file.
The same 20 MB took 18 seconds as small files and 28 milliseconds as one file. That is several hundred times slower, for identical bytes. On Windows the gap is especially large, because antivirus software also inspects each file as it is opened. On Linux the gap is smaller, but it is still large.
Every small file costs a path lookup, an inode read, an open, a read and a close, and each of those is a system call (2.1.2) that may touch a different place on the disk. One big file costs one of each, then streams. This page uses that fact to answer a real curiosity: why a tool called WizTree can measure a whole disk in seconds while Windows Explorer takes minutes. On the way, it tours the file systems you actually use, explains how Linux makes one read call work on all of them, and shows how the physical disk underneath changes the rules.
1. The file systems you will meet
Everything in 2.6.1 and 2.6.2 is common to all of these. They differ in their details and in what they were built for.
| File system | Used by | Crash safety | Notable for |
|---|---|---|---|
| ext4 | most Linux machines | journal | extents, mature, everywhere |
| XFS | big Linux servers | journal | large files, parallel writes |
| NTFS | Windows | journal | the Master File Table, access lists |
| APFS | macOS, iPhone | copy-on-write | snapshots, instant copies |
| Btrfs, ZFS | Linux servers, storage boxes | copy-on-write | snapshots, checksums on all data |
| FAT32, exFAT | USB sticks, SD cards | none | readable by every device |
ext4
ext4, the "fourth extended file system", became stable in 2008 and is the default on most Linux distributions. It is the inode design of 2.6.1 with the journal of 2.6.2, plus one big improvement: extents.
Instead of a pointer for every block, an inode records ranges: "this file's data is blocks 50,000 to 52,047". One extent can describe up to 128 MB of continuous data. A 1 GB video stored in eight long runs needs eight extents, instead of the quarter of a million block pointers the old scheme needed. Lookups get faster, and the file system is encouraged to keep files in long continuous runs, which spinning disks read much faster (section 5).
ext4 also uses delayed allocation: it waits until data is about to be written before choosing which blocks to put it in, so it can pick one long run instead of scattered blocks. That was the change that exposed the zero-length-file bug in 2.6.2.
NTFS
NTFS, the "New Technology File System", arrived with Windows NT in 1993 and is still Windows's main file system. It journals its metadata, supports hard links (which is why the inode demo in 2.6.1 worked on Windows), and uses access control lists for permissions instead of rwx bits (2.9.4).
Its central structure is the Master File Table, or MFT: one table with a record, usually 1 KB, for every file and folder on the volume. Each record holds the file's name, its size, its timestamps, its parent folder and where its data lives. The MFT plays the part of the inode table, with one difference worth noting: the name is stored in the file's own record too, along with a reference to its parent folder. Very small files, a few hundred bytes, are even stored inside their MFT record, with no separate data blocks at all.
APFS
APFS, the Apple File System, replaced Apple's older file system in 2017 on iPhones and Macs. It is built for SSDs and uses copy-on-write (2.6.2). Copying a file in Finder is instant because APFS makes a clone that shares blocks with the original. Time Machine backups use APFS snapshots. Encryption is built in. By default on a Mac it ignores upper and lower case, so Beach.jpg and beach.jpg are the same name, which is a classic source of bugs when code moves to a Linux server where they are different (2.9.1).
FAT32 and exFAT
FAT, the file allocation table, dates from 1977 and is the simplest of all. It has no journal, no permissions and no hard links. Its strength is that almost every device ever made can read it: cameras, cars, televisions, printers. That is why USB sticks and memory cards still use it. Its best-known limit is that FAT32 cannot hold a single file of 4 GB or more, which is why copying a long video to an old USB stick fails with a "file too large" error. exFAT, from 2006, removes that limit while staying simple, and is the standard for large memory cards.
FAT is also the file system the UEFI firmware can read, which is why the EFI System Partition from 2.1.3 is formatted with it.
2. Why WizTree scans a whole disk in seconds
Here is the curiosity. WizTree is a Windows tool that shows what is using space on a disk. It can scan a multi-terabyte drive with millions of files in a few seconds. Windows Explorer's "Properties" on a big folder can take minutes to add up the size of the same files. Why such a difference? ⚑What is WizTree and how does it work? [EQ-79]
Explorer works the way the opening demo's line (3) did. It walks the folder tree. For each folder, it lists the names. For each name, it asks for that file's details. A million files means a million separate requests, each a system call, each a small read from a different place on the disk. That is exactly the slow "20,000 small files" row, multiplied by fifty.
WizTree reads the Master File Table directly. With administrator rights, it opens the volume itself and reads the MFT as one large file, in big sequential chunks. Every file's record is in there, with its name, its size and a reference to its parent folder. So in one long read of a few gigabytes, WizTree has every file's size and location. It then rebuilds the whole folder tree, and every folder's total, in memory, which takes moments.
The lesson travels far beyond one tool. How you read data, one big sequential read or many small scattered ones, can matter far more than how much data there is. It is the same locality principle as CPU caches (1.6), now at the scale of a whole disk. The search tool Everything uses the same trick to find any file on a Windows disk instantly. And the same lesson is why backups ship one big archive file rather than millions of little files (section 6).
3. The VFS: one interface for every file system
Your programs call open, read, write and close. They never ask whether the file lives on ext4, on a FAT32 USB stick, on a network share, or somewhere stranger still. That is possible because of a layer inside the kernel called the virtual file system, or VFS.
The VFS defines a fixed list of operations that every file system must provide: "look up a name in a directory", "read bytes from a file", "write bytes", "list a directory", and so on. Each real file system supplies its own versions of those operations. When the upload service calls read, the system call reaches the VFS, which finds out which file system the file belongs to and calls that file system's read operation.
Each file system is attached to a place in the single tree of folders by mounting it. The main disk is mounted at /. A USB stick might be mounted at /media/usb. From then on, any path under /media/usb is handled by the USB stick's file system. Windows does the same thing with drive letters, C:, D: and so on, instead of folders.
This is the same design idea as an interface with several implementations in your own code (9.2.5), applied inside the kernel. And it has a remarkable consequence. A "file system" does not need a disk at all. It only needs to answer the VFS's questions.
/procis invented by the kernel as you read it. Each "file" in it describes the live state of the system or of a process. Reading/proc/meminfomakes the kernel generate a description of memory use at that moment. 2.8.3 explores it.tmpfskeeps its files only in RAM. They are very fast and vanish at reboot. The/dev/shmfolder that held the named semaphore in 2.4.2 is atmpfs.- Network file systems, such as NFS on Linux and SMB on Windows (2.9.4), turn each
readinto a request sent to another machine. - FUSE, "file system in user space", lets an ordinary program answer the VFS's questions. Tools built on it can make a remote server, or cloud storage, look like a local folder.
This is the famous Unix saying, everything is a file. Disks, devices, processes, kernel settings and network shares all appear through the same open, read and write, so the same tools work on all of them.
4. One more layer down: the disk itself
File systems were designed around the physics of the device underneath, and the two kinds of device behave very differently.
Spinning hard disks: everything is about seeks
A hard disk drive, or HDD, stores data on spinning metal platters, read by a head on a moving arm. To read a block, the arm must move to the right track, which is called a seek, and then wait for the block to spin round under the head. A seek takes several milliseconds. At 7,200 revolutions per minute, one rotation takes about 8.3 ms, so waiting for the block takes about 4 ms on average.
So a hard disk can do only about 100 to 200 random reads a second. But once the head is in place, it can read continuous data at well over 100 MB a second. Random access is hundreds of times slower than sequential access. That is why file systems try so hard to keep each file in long continuous runs, why extents help, and why the "many small files" demo is so much worse on a hard disk.
Solid-state drives: no seeks, new rules
A solid-state drive, or SSD, stores data in flash memory chips with no moving parts. A random read takes tens of microseconds, so an SSD can do tens of thousands to hundreds of thousands of random reads a second. 2.7.4 calls that number IOPS.
But flash memory brings its own strange rules.
- It cannot overwrite in place. Data is written in pages of a few kilobytes, but can only be erased in much larger erase blocks, often hundreds of kilobytes to megabytes. To change one page, the drive writes the new version to a fresh page elsewhere and marks the old one as stale.
- Each cell wears out after a limited number of erase cycles.
To hide all this, every SSD contains its own small computer running a program called the flash translation layer. It keeps its own map from the block numbers the operating system uses to the physical pages where the data really is. It also does wear levelling, spreading writes evenly so no part of the chip wears out early, and garbage collection, copying the still-valid pages out of an erase block so the whole block can be erased and reused.
Here a problem appears. When you delete a file, the file system marks its blocks free in its own bitmap (2.6.1). But the SSD does not know that. To the SSD, those pages still hold valid data, and its garbage collection keeps carefully copying them around. The fix is the TRIM command. The file system tells the SSD "these block numbers no longer hold anything you need", so the drive can discard them. Modern operating systems send TRIM automatically.
Copy-on-write file systems and the log-structured storage engines of databases (7.3.3) fit flash memory well, because they turn scattered small overwrites into new writes in fresh places, which is what flash prefers anyway. How SSDs connect to the CPU, and why some are ten times faster than others that look the same, is in 2.7.1.
5. When many small files are the problem
Back to the opening demo. Whenever a workload consists of very many small files, the cost is dominated by per-file overhead, not by the data. The standard fixes all turn many small operations into few large ones:
- Pack them into one file. Archive tools such as
tarandzipturn millions of files into one, which can be copied or uploaded as a single long stream. This is why backups and software releases ship archives. - Use the file system's own tools. Snapshots and clones on copy-on-write file systems copy millions of files in an instant, by sharing blocks instead of reading and writing them.
- Store small things somewhere better suited. Thousands of tiny records are often better in a database, or as keys in an object store, than as files.
What the interviewer will push on
"Why can WizTree scan a disk in seconds while Explorer takes minutes?" They are checking that you understand access pattern, not just the tool. Explorer walks the tree with a stat per file: millions of small, scattered requests. WizTree reads NTFS's Master File Table, which already holds every file's name, size and parent, in one long sequential read, and builds the totals in memory.
"What is the VFS?" A layer in the kernel that defines one set of file operations and sends each call to the right file system's implementation. The strong answer gives the consequence: things that are not on disk, such as /proc, tmpfs and network shares, can look like files.
"How are SSDs different from hard disks, and what is TRIM?" Hard disks are dominated by seek time, so random access is hundreds of times slower than sequential. SSDs have fast random access but cannot overwrite in place and wear out, so a flash translation layer remaps writes and levels wear. TRIM tells the drive which blocks the file system has freed.
"Why are millions of small files slow to copy, and how would you speed it up?" Per-file overhead: path lookups, inode reads and several system calls per file, plus scattered disk access. Pack them into an archive, use snapshots or clones, or store small items in a database.
"What are extents?" Ranges of continuous blocks recorded as one entry, instead of one pointer per block. They shrink metadata for large files and encourage continuous layout.
Something to volunteer that nobody asks for: case sensitivity differs between file systems. macOS and Windows normally treat Beach.jpg and beach.jpg as the same name; Linux treats them as different. Code that works on a laptop and fails on a Linux server because of an import with the wrong case is a classic, and naming it shows experience.
Recall
- ext4 on Linux uses inodes, a journal and extents. NTFS on Windows keeps every file's record, including its name and parent, in the Master File Table. APFS, Btrfs and ZFS use copy-on-write. FAT32 and exFAT are simple and readable everywhere, and FAT32 cannot hold a 4 GB file.
- WizTree reads the MFT in one long sequential read and rebuilds every folder's size in memory, while Explorer makes one request per file. Access pattern beats data size.
- The virtual file system gives every file system the same operations and dispatches each call; file systems are mounted into one tree.
/proc,tmpfs, network shares and FUSE show that "everything is a file". - Hard disks pay a seek of milliseconds, so random access is hundreds of times slower than sequential. SSDs have fast random access but erase in large blocks and wear out, hidden by a flash translation layer with wear levelling; TRIM tells the drive which blocks are free.
- Many small files are slow because of per-file overhead; pack them into archives, use snapshots, or use a database.
Self-test: Why did 20,000 small files take hundreds of times longer than one 20 MB file? What does an NTFS MFT record hold that lets WizTree avoid opening any folders? What happens, step by step, when a program calls read on a file on a USB stick? Why can a hard disk do only a hundred or so random reads a second? Why does an SSD need TRIM?
Quiz Bank
FoundationalWhat is the VFS?
The virtual file system is an abstraction layer inside the kernel. It defines one standard set of operations, such as look up a name, read, write and list a directory, and each real file system provides its own implementation of them. A program's read system call reaches the VFS, which calls the read operation of whichever file system holds that file. So the same programs and tools work on ext4, FAT, network shares and more, and things that are not stored on any disk, such as /proc, can appear as files.
FoundationalWhy do USB sticks still use FAT32 or exFAT, and what is FAT32's best-known limit?
Because FAT-family file systems are simple and supported by almost every device ever made, including cameras, cars, televisions and the UEFI firmware of PCs. Richer file systems such as ext4, NTFS or APFS are not readable everywhere. FAT32's best-known limit is that a single file must be smaller than 4 GB, which is why copying a large video to an old stick fails. exFAT removes that limit while staying simple, and is the standard for large memory cards.
AppliedWhy can WizTree scan a whole drive in seconds while Explorer takes minutes to total a folder?
Explorer totals a folder by walking the tree: listing each folder and asking for each file's size, which means millions of small, scattered requests and system calls. WizTree instead reads NTFS's Master File Table directly, which holds a record for every file containing its name, size and parent folder. It reads that one table in large sequential chunks, then rebuilds the whole tree and every folder's total in memory. One long sequential read beats millions of small random ones by orders of magnitude: the access pattern matters more than the amount of data.
InterviewHow do SSDs differ from hard disks, and how does that affect software?
A hard disk must move its head to a track and wait for the platter to rotate, so each random access costs milliseconds and it manages only about 100 to 200 random reads a second, while sequential reads are fast. Software designed for hard disks keeps data continuous and avoids random access.
An SSD has no moving parts, so random reads take tens of microseconds and it can do tens of thousands or more a second. But flash cannot be overwritten in place, is erased in large blocks, and wears out, so the drive's flash translation layer remaps writes, levels wear and garbage-collects stale pages. Software benefits from avoiding small scattered overwrites, which cause extra internal copying, and from sending TRIM so the drive knows which blocks are free. Copy-on-write and log-structured designs suit SSDs well.
InterviewA backup script copying millions of tiny files is far slower than copying one large file of the same total size. Explain and suggest faster approaches.
Each small file costs a fixed overhead: resolving its path, reading its inode, opening, reading and closing it, and at the destination creating a new inode and directory entry and updating bitmaps and the journal. These are separate system calls touching scattered places on the disk, so with millions of files the overhead dwarfs the data. On a hard disk, the scattered access adds a seek to many of them.
Faster approaches:
- Pack the files into one archive with
tarorzip, and copy that as one stream. - Use tools that batch the work and run it in parallel.
- On copy-on-write file systems, use snapshots or clones, which copy millions of files instantly by sharing blocks. Longer term, store large numbers of tiny items in a database or object store rather than as individual files.
StaffA team moves its build from Linux laptops to Mac laptops and some builds fail with missing files that clearly exist. Others fail only on the Linux build servers. What is the likely cause, and how should they prevent it?
Case sensitivity. APFS on macOS, like NTFS on Windows, normally ignores the difference between upper and lower case, while Linux file systems treat Photo.ts and photo.ts as different names. Code that imports ./Photo when the file is photo.ts works on a Mac or Windows machine and fails on Linux. The reverse also happens: two files differing only in case can coexist in a Linux repository but collide on a Mac, so one silently replaces the other.
Prevent it by making the build environment match production, for example building in the same Linux container image everywhere, by having linting or compiler settings that reject imports whose case does not match the file on disk, and by forbidding file names that differ only in case.
Scenario Drill
DrillThe photo-upload service stores millions of 1 KB thumbnails as individual files on one server. Backups take all night, and listing a folder takes seconds. Using this page, redesign the storage.
The problem is per-file overhead. Each thumbnail costs a directory entry, an inode, and several system calls to create, read, back up or list, and folders with hundreds of thousands of entries are slow to list. The data itself is small; the bookkeeping is not.
Options, best first:
- Pack thumbnails into larger files. Store thumbnails in batches inside large container files, with a small index recording where each one sits. Reads become a seek within one big file, and backups copy a few large files in long sequential streams.
- Use an object store designed for millions of small objects, keyed by photo ID, which handles distribution, durability and listing for you.
- At least spread the files over many subfolders, for example by the first characters of a hash of the ID, so no single folder holds a huge number of names.
For backups specifically, a copy-on-write file system's snapshots, or an archive written as one stream, avoid touching each file individually. The general rule from this page applies: turn many small operations into few large ones.
Next: we have treated the disk as something that simply answers when asked. But the CPU is millions of times faster than any disk or network card. 2.7.1 asks what the CPU does while it waits, how a device tells the CPU it has finished, and how data moves into memory without the CPU copying every byte.