Skip to content

Volume ITechnology

2.6.2 — Crash Safety & Durability ​

The photo-upload service keeps an index file, photo-index.json, listing every photo it has saved. After each upload it writes the file again. One night the power fails while it is writing. In the morning, the index file is half old and half new, and the service cannot read it at all. Twelve hundred photos are on the disk, and the service has no idea they exist.

Two separate problems hide in that story.

  1. Durability. When writeFile returns, has the data actually reached the disk? Usually, no.
  2. Atomicity. Can a change made of several writes happen all at once, so that a crash leaves either the old version or the new one, and never a broken mixture?

The file system faces both problems for its own structures, the inodes, bitmaps and directories of 2.6.1. Your programs face both for their own files. This page explains how file systems solve them, and how you solve them in your own code with three system calls and a rename.

1. Why one save is many writes ​

2.6.1 showed that saving a new photo touches several separate places on the disk. Creating album/beach.jpg needs at least five writes:

  1. mark an inode as used in the inode bitmap;
  2. fill in the inode: size, owner, block pointers;
  3. mark the data blocks as used in the block bitmap;
  4. write the photo's bytes into the data blocks;
  5. add beach.jpg → inode to the directory's table.

The disk performs these one at a time, and the kernel may even send them in a different order, to save time. Now imagine the power fails after some of them but not others.

Done before the crashWhat is wrong afterwards
1–3 onlyblocks and an inode marked used, but no file uses them: space leaked
5 before 2a name points at an inode full of garbage
2 before 4a file of the right size containing old data from someone else's file

The last row is the dangerous one. Blocks freed by some other deleted file may still hold that file's contents. A crash at the wrong moment could give the new file someone else's old data. A file system left in states like these is called inconsistent.

2. The old fix: check the whole disk ​

For decades, the fix was to scan the entire file system after any crash. A program called fsck, for file system check, read every inode, every directory and every bitmap, and made them agree. It freed leaked blocks, removed names pointing at nothing, and put files with no name into a folder called lost+found.

It worked, but its cost grew with the size of the disk. On a large server disk it could take hours, during which the server was down. As disks grew from megabytes to terabytes, this became unacceptable.

3. Journaling: write down the plan first ​

The modern fix comes from databases, which had the same problem first (7.3.3). It is called journaling, and the idea is the one you use when doing something risky by hand: write down what you are about to do, before you do it.

The file system sets aside a region of the disk called the journal, also known as a write-ahead log. To make a change made of several writes, it works in four steps.

  1. Write the plan to the journal. Record every block that is about to change, with its new contents: "the bitmap block becomes this, the inode becomes this, the directory block becomes this".
  2. Write a commit record. One final, small record saying "this group of changes is complete". Writing one small record is effectively atomic: it is either fully on disk or not at all.
  3. Apply the changes to their real places on the disk. This step is called checkpointing.
  4. Mark that journal entry as finished, so its space can be reused.

Now consider a crash at every possible moment.

  • Before the commit record is written: the journal holds a plan with no commit. On restart, the file system throws the half-written plan away. The real blocks were never touched, so the disk is exactly as it was before. The change simply did not happen.
  • After the commit record, before or during step 3: the journal holds a complete, committed plan. On restart, the file system replays it, writing every block again. Writing a block twice does no harm, so it does not matter how much of step 3 had finished. The change happens completely.

Either way, after a restart, the file system is consistent. And recovery only reads the journal, which is small, instead of the whole disk. Recovery now takes seconds instead of hours.

1journal"inode, bitmap, dir become…"2commitone small record3apply in placethe real blocks4clearentry donecrash before ②discard the plan: as if it never happenedcrash after ②replay the plan: it happens completely
Figure 1 — Journaling. ① The plan is written to the journal. ② A single commit record marks it complete. ③ The changes are applied in place. ④ The entry is cleared. A crash before the commit discards the plan; a crash after it replays the plan. There is no third outcome.

The cost is that changed blocks are written twice: once into the journal and once into place. To keep that cost down, most file systems by default journal only the metadata, meaning the inodes, bitmaps and directories, and not the file data itself. Linux's ext4 has three settings:

ModeWhat goes in the journalEffect
journalmetadata and datasafest, slowest
ordered (default)metadata only, data written firstno garbage in new files
writebackmetadata only, no orderingfastest, may show old data

The default, ordered, closes the dangerous row of the table in section 1. It makes sure a file's data blocks are on disk before the metadata that points at them is committed, so a crash can never leave a new file pointing at someone else's old data. But because data is not in the journal, the file system's own structure is protected, and your file's contents are not. If the program was halfway through rewriting photo-index.json, the file system will be perfectly consistent, and the file will still be half old and half new. That is your job, in section 6.

4. Copy-on-write: never overwrite anything ​

There is a second way to get atomic changes, and newer file systems use it: Apple's APFS, and Btrfs and ZFS on Linux. It is copy-on-write, the same idea as copy-on-write fork in 2.2.2. Never overwrite a block in place. Write the new version somewhere free, then switch one pointer to it.

The whole file system is a tree of blocks, with one root block at the top that leads to everything else. To change a file, a copy-on-write file system:

  1. writes the changed data blocks to free space;
  2. writes new copies of the blocks that point at them, all the way up the tree, also to free space;
  3. finally, writes a new root that points to the new tree.

Until step 3, the old root still describes the old file system, completely and correctly, because nothing it points at was changed. Step 3 is one small write. A crash before it leaves the old version. A crash after it leaves the new one. There is no journal to replay, because the disk is always consistent.

Copy-on-write gives two features almost for free.

  • Snapshots. Keep the old root instead of freeing it, and you have a complete, read-only picture of the file system at that moment. It costs nothing up front, because the old and new trees share every block that has not changed since. Apple's Time Machine and many backup systems use this.
  • Instant copies. Copying a 10 GB file can just create a second file pointing at the same blocks. Blocks are copied only when one of the files is changed. On macOS this happens automatically when you copy a file in Finder on APFS.

The costs: writes are scattered around the disk, and a file that is changed a little at a time, such as a database file, becomes spread across many places over time.

5. The page cache: why write returns before the data is safe ​

Now the durability problem. When the upload service calls writeFileSync, the kernel does not wait for the disk. It copies the bytes into the page cache (2.5.4), marks those pages dirty, and returns at once. The actual disk writes happen later, in the background. On Linux, dirty data is normally written out within about 30 seconds.

This is a large part of why computers feel fast. Writes complete at memory speed, and many small writes to the same file are combined into a few big ones. Reading a file you just wrote is instant too, because it is still in the cache.

But it means something important. When write returns, the data is in RAM, not on the disk. If the power fails in the next few seconds, it is gone, even though every call reported success.

To make sure data has really reached the disk, a program must call fsync on the file. fsync does not return until the kernel has written the file's dirty pages and its metadata to the device, and told the device to flush its own internal cache. Disks have caches too, and a disk that has only cached a write could lose it in a power cut just like RAM. A cheaper variant, fdatasync, skips metadata that does not affect reading the data back, such as the last-modified time.

fsync is slow by comparison, because it waits for real hardware. On the SSD used for this book's measurements, a durable save took about 2 ms, against about 0.4 ms for a plain write (section 6 shows the code). On a spinning hard disk it can be 10 ms or more. That cost is why databases put a great deal of effort into calling fsync as rarely as they safely can, for example by committing many transactions with a single fsync, called group commit. The "D" in a database's ACID promise, durable, rests directly on this call (7.4.1).

6. Saving a file safely: the atomic-save pattern ​

Here is the fix for the broken index file. It uses three facts:

  • fsync makes data durable;
  • rename within one file system is atomic: other programs see either the old name pointing at the old inode, or the same name pointing at the new inode, never a mixture (2.6.1 showed rename only edits one directory entry);
  • a directory is a file too, so the rename itself becomes durable only when the directory is fsynced.
ts
// atomic-save.ts — save the photo index so a crash leaves the old version or the new one, never half
import { openSync, writeSync, fsyncSync, closeSync, renameSync, readFileSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";

function saveAtomically(path: string, data: string): void {
  const tmp = `${path}.tmp-${process.pid}`;                    // (1) a temporary name beside it
  const fd = openSync(tmp, "w");
  writeSync(fd, data);                                         // (2) write the whole new version
  fsyncSync(fd);                                               // (3) force it onto the disk
  closeSync(fd);
  renameSync(tmp, path);                                       // (4) swap it in, in one step
  if (process.platform !== "win32") {                          // (5) make the rename itself durable
    const dir = openSync(dirname(path), "r");
    fsyncSync(dir);
    closeSync(dir);
  }
}

saveAtomically("photo-index.json", JSON.stringify({ photos: 1200 }));
saveAtomically("photo-index.json", JSON.stringify({ photos: 1201 }));
console.log(readFileSync("photo-index.json", "utf8"));

// What durability costs: 200 small saves with and without fsync
let t = performance.now();
for (let i = 0; i < 200; i++) writeFileSync(`plain-${i}.json`, "{}");
const plain = performance.now() - t;
t = performance.now();
for (let i = 0; i < 200; i++) saveAtomically(`safe-${i}.json`, "{}");
const safe = performance.now() - t;
console.log(`200 plain writes: ${plain.toFixed(0)} ms   200 durable saves: ${safe.toFixed(0)} ms`);

// Printed on a laptop SSD (a second run gave 63 ms and 406 ms):
// {"photos":1201}
// 200 plain writes: 93 ms   200 durable saves: 400 ms
  • (1) Never write over the real file. Write a new one beside it, in the same directory, so the later rename stays within one file system. The process ID in the name stops two copies of the service from using the same temporary file.
  • (2) Write the complete new version. If the program crashes here, the real photo-index.json is untouched, and only a stray temporary file is left behind.
  • (3) fsync the temporary file, so its contents are truly on the disk before it gets the real name. Skipping this step is the most common mistake. Without it, the rename could reach the disk before the data does, and a crash would leave photo-index.json pointing at an empty or half-written file.
  • (4) rename swaps the new file in. Any reader, at any instant, sees either the complete old index or the complete new one.
  • (5) On Linux and macOS, fsync the directory, because the rename changed the directory's table and that change also sits in the page cache until it is flushed. Windows does not allow opening a directory this way, so the step is skipped there.

The measured cost, about 2 ms per save against well under 1 ms, is the price of durability. For an index saved after every upload, it is a good price. For a log line written a thousand times a second, it is not, and a log would instead fsync in batches, accepting that the last moment of logs could be lost in a crash.

This pattern is how careful programs save configuration files, how text editors save documents, and how package managers update files. When you see write temp, fsync, rename, fsync dir in someone's code, this is what they are doing.

A real story: zero-length files ​

In 2009, Linux's ext4 file system started holding newly written data in memory for longer before choosing where on the disk to put it. That made writes faster. It also exposed a large number of programs that did steps (1), (2) and (4) but skipped step (3). After a crash, users found many configuration files completely empty: the rename had reached the disk, but the data had not. The programs had always been wrong. Older file systems had just happened to write the data first. ext4 added a safety rule that forces the data out early when it sees this pattern. The lesson stands: if your code needs data to survive a crash, call fsync; never rely on the file system happening to write in a helpful order.

What the interviewer will push on ​

"What is journaling, and what problem does it solve?" They are checking whether you can explain the crash at each moment. A good answer has three parts. Changes are written to a journal, then a commit record, then applied in place. A crash before the commit discards the plan, and a crash after it replays the plan. So recovery reads only the journal, in seconds, instead of checking the whole disk. The tell is mentioning that most file systems journal only metadata, so your file's contents are not protected.

"write returned success, then the power failed, and the data is gone. Why?" The page cache. write only copies data into RAM and returns; the kernel writes it to disk later. Durability needs fsync, which also flushes the disk's own cache.

"How do you update a file so a crash cannot corrupt it?" The atomic-save pattern: write a temporary file in the same directory, fsync it, rename it over the original, then fsync the directory. Explain why each step is there, especially the first fsync.

"What is a copy-on-write file system, and what does it give you?" It never overwrites blocks; it writes new versions and switches one root pointer, so it is always consistent. That gives cheap snapshots and instant file copies.

"Why are databases so careful about fsync?" Because it is what makes a committed transaction durable, and it is slow. Group commit makes many commits share one fsync.

Something to volunteer that nobody asks for: in 2018 PostgreSQL's developers found that on Linux, after fsync reports an error, the kernel may mark the failed pages as clean. Retrying fsync then reports success, although the data never reached the disk. PostgreSQL changed to stop immediately on any fsync error instead of retrying. It shows that durability depends on handling errors from fsync correctly, not just on calling it.

Recall

  • One save is several separate writes (bitmaps, inode, data, directory), so a crash can leave the file system inconsistent. The old fix, fsck, scanned the whole disk and could take hours.
  • Journaling writes the plan and a commit record first, then applies it: a crash before the commit discards the plan, after it replays it, so recovery takes seconds. ext4's default ordered mode journals metadata only and writes data first, so the file system is safe but your file's contents are not.
  • Copy-on-write file systems (APFS, Btrfs, ZFS) never overwrite; they write new blocks and switch one root pointer, giving cheap snapshots and instant copies.
  • write returns once data is in the page cache; only fsync waits for the disk, including its cache. It is slow, about 2 ms against 0.4 ms in our measurement, so databases batch it with group commit.
  • The atomic-save pattern: write a temporary file, fsync it, rename it over the original (atomic), then fsync the directory. Skipping the first fsync can leave an empty file after a crash.

Self-test: Name the writes involved in creating a file. Explain what happens on restart if the crash came just before, and just after, the commit record. Why does ext4's default mode still not protect a half-rewritten file? Why must the temporary file be fsynced before the rename? What does a copy-on-write file system write last, and why does that make it crash-safe?

Quiz Bank ​

FoundationalWhat is journaling and what problem does it solve?

Creating or changing a file needs several separate block writes, such as bitmaps, an inode, data and a directory entry, and a crash between them leaves the file system inconsistent. Journaling writes a description of all the changes to a dedicated area, the journal, followed by a commit record, and only then applies them in place. After a crash, a plan without a commit is discarded and a committed plan is replayed, so the file system is always consistent. Recovery reads only the journal, taking seconds, instead of scanning the whole disk with fsck, which could take hours.

FoundationalWhy can data be lost after write() returns success?

Because write normally only copies the data into the kernel's page cache in RAM, marks the pages dirty, and returns; the kernel writes dirty pages to the disk later, typically within about 30 seconds. A power failure before then loses the data even though write succeeded. To guarantee the data is on the disk, the program must call fsync (or fdatasync), which returns only after the data and metadata have been written and the device has been told to flush its own cache.

AppliedHow do you update a file so that a crash leaves either the old or the new version, never a mixture?
  1. Write the complete new contents to a temporary file in the same directory.
  2. Call fsync on the temporary file, so its data is durable before anything refers to it.
  3. rename the temporary file over the original. Within one file system rename is atomic: readers see the whole old file or the whole new one.
  4. Call fsync on the directory, so the rename itself is durable.

A crash before step 3 leaves the original untouched. A crash after step 4 leaves the new version. Skipping step 2 is the common bug: the rename can reach the disk before the data, leaving an empty or partial file.

InterviewWhat is a copy-on-write file system, and what does it give you?

A copy-on-write file system never overwrites a block in place. To change data, it writes new copies of the changed blocks and of every block above them in its tree, all into free space, and finally writes a new root pointer. Until that last small write, the old root still describes a complete, consistent file system; after it, the new root does. So crashes never leave it inconsistent, and no journal replay is needed.

It also makes snapshots nearly free, since keeping an old root keeps a complete earlier picture that shares all unchanged blocks, and instant copies, where a copied file shares blocks with the original until one is changed. APFS, Btrfs and ZFS work this way. The cost is more scattered writes and fragmentation for files changed in small pieces.

InterviewYour application calls write() and returns success, but after a power loss some saved data is missing. Explain why and how to guarantee durability.

write only placed the data in the page cache; the kernel had not yet written it to disk when the power failed. Most file systems' journals protect their own structure, not file contents, so the file system was consistent but the data was simply never written.

To guarantee durability, call fsync on the file after writing, and on the directory when creating or renaming files, so the name is durable too. Accept the cost: it waits for the hardware, milliseconds per call. For frequent writes, batch them and fsync once per batch, as databases do with group commit. Also handle fsync errors as fatal: on Linux, a failed fsync may leave the kernel believing the pages are clean, so retrying can falsely report success.

StaffA team stores each user's settings as a JSON file and, after a crash, some users' settings files are empty. They currently open the file with truncation and write the new JSON. Explain the failure and design a fix, including its cost.

Opening with truncation empties the file immediately; the new JSON is then written into the page cache. A crash after the truncation reached the disk but before the new data did leaves an empty file. With delayed allocation the gap can be seconds long. Even without a crash, a reader at the wrong moment sees an empty or partial file.

The fix is the atomic-save pattern: write settings.json.tmp, fsync it, rename it over settings.json, then fsync the directory. Every reader and every crash then sees a complete old or new file. Clean up stray temporary files at startup.

The cost is two fsync calls per save, a few milliseconds on an SSD. For settings that change rarely, that is negligible. If saves were very frequent, they could be combined per user with a short delay, or the settings moved into a database, which provides atomic, durable updates with its own write-ahead log.

Scenario Drill ​

DrillThe photo-upload service writes each uploaded photo with writeFile and immediately replies 'saved' to the user. After a power cut, a few photos that users were told were saved are missing or truncated. Using this page, explain and fix it, and say what it costs.

The reply was sent as soon as writeFile returned, which only means the photo was in the page cache. The power cut came before the kernel wrote those pages to disk. So users were told "saved" about data that existed only in RAM.

The fix depends on what "saved" must mean. If it must mean "will survive a crash":

  1. Write each photo to a temporary name.
  2. fsync it.
  3. rename it to its final name.
  4. fsync the directory.
  5. Only then reply "saved".

Any photo the service acknowledged is then on disk, and no reader ever sees a half-written photo under its real name.

The cost is a few milliseconds of fsync per upload, small compared with receiving a multi-megabyte photo over the network. Under heavy load, the service can fsync several finished photos together before acknowledging them all, which is group commit applied to files. If photos are also copied to a second machine or to cloud storage, durability can be defined as "stored in two places", which protects against a failed disk as well as a power cut.

Next: everything so far described "the file system" in general terms. Page 2.6.3 looks at the real ones you use, ext4, NTFS, APFS and the FAT on a USB stick, answers why a tool like WizTree can scan a whole drive in seconds, and shows how Linux lets one read call work on all of them, and on things that are not files at all.

Combine chapters into one PDF

Pick the chapters you want. They print in the order you pick them.

Volume I — Technology
Part 0 — Retention & Notation
Part 1 — Foundations of Computation
Part 2 — Operating Systems
2.1 The Kernel
2.2 Processes
2.3 Threads & Scheduling
2.4 Concurrency Under the Hood
2.5 Memory Management
2.6 File Systems
2.7 Input/Output
2.8 Linux for Engineers
2.9 Windows & macOS in Practice
Part 3 — Languages, Compilers & Runtimes
3.6 JavaScript & V8
3.7 TypeScript
3.8 Node.js Internals
3.13 Python & CPython
3.14 FastAPI & Production Python
Part 4 — Data Structures & Algorithms
4.0 Classic Problems — Before NeetCode
4.4 Arrays & Hashing — Problems
4.5 Two Pointers
4.6 Sliding Window
4.8 Stack — Problems
4.9 Linked List — Problems
4.11 Binary Search — Problems
4.13 Trees
4.14 Tree — Problems
4.15 Trie — Problems
4.17 Heap — Problems
4.18 Backtracking — Problems
4.19 Graphs
4.20 Graph — Problems
4.21 Advanced Graph — Problems
4.23 1-D DP — Problems
4.24 2-D DP — Problems
4.26 Greedy — Problems
4.27 Interval — Problems
4.28 Math & Geometry — Problems
4.30 Bit Manipulation — Problems
Part 5 — Networking & the Web
5.3 IP
5.4 Transport
5.6 HTTP
Part 6 — Frontend Engineering
6.1 Browser Internals
6.2 HTML & CSS Mastery
6.3 JavaScript in the Browser
6.4 React, from First Component to Full Shop
6.5 Frameworks & Rendering
6.8 UI Engineering & Patterns
Part 7 — Databases & Storage
7.2 SQL Mastery
7.3 Storage Engine Internals
7.4 Transactions
7.5 NoSQL Taxonomy
7.8 Analytics & Big Data
Part 8 — Security, Identity & Cryptography
8.2 Cryptography
8.3 Protocols of Trust
8.4 Identity: AuthN & AuthZ
8.5 Application Security
8.6 Infrastructure Security
Part 9 — Software Design (LLD)
9.2 OOP Foundations
9.3 Design Principles
9.4 Design Patterns
9.5 Concurrency
9.6 API Design
9.7 LLD Interview Machines
9.9 Node/Express Production Playbook
Part 10 — Distributed Systems & HLD
10.7 Consistency & Consensus
10.8 Messaging & Microservices
10.14 Caching
Building blocks from the design seat
Part 11 — System Design Case Studies
Part 12 — AI & ML Engineering
12.1 What AI Engineering Is
12.2 The Math You Need
12.3 Classical Machine Learning
12.4 Data for ML
12.5 Deep Learning
12.6 Embeddings & Vector Search
12.7 Transformers & LLMs
12.8 Building with LLMs
12.9 Retrieval-Augmented Generation
12.10 Agents
12.11 Adapting Models
12.12 Production AI
12.13 Multimodal AI
12.14 AI Security & Governance
12.15 Case Studies & Capstone
Part 13 — Cloud, DevOps & Infrastructure
13.11 Git, from First Commit to Team Workflow
Part 14 — Containers: Docker & Kubernetes
14.1 Foundations
14.2 Docker
14.3 Kubernetes
14.4 Kubernetes on Azure
Part 15 — The Craft & Career
Nothing picked yet