16Durability — flush, fsync, and the page cache

In Chapter 15 we opened a file, handed it bytes, and closed it. Python let us walk away believing the data was safe on disk. It wasn't. In this chapter we chase those bytes all the way down to the metal. We find the exact moment "saved" becomes true, because there's a lie hiding in write() that most tutorials never make you look at. Here's the plan. First we catch write() returning a number before a single byte has touched the platter. Then we peel apart the three volatile buffers stacked between your string and the disk — Python's own, the kernel's, and the drive's. And we meet os.fsync, the one syscall that forces bytes past all of them. The whole way down we keep asking the one thing that actually matters: if the power cut right now, which of my writes would survive? By the end we've built the atomic write-temp-then-rename save that never leaves a half-written file. We've also watched how a database pays fsync's brutal price just once for a thousand records. "I saved it" stops being a hope and becomes a claim you can prove.

01write() lies: the page cache and write-back

Let's start by watching the lie happen. You wrote the code. f.write(record) returned 128, the length of your line, right there in the return value. The next statement, print("saved!"), ran, and you watched it print. Then the building lost power for one second. The machine rebooted, you reopened the file, and the record is gone. Not garbled. Not half-there. Gone, as though write() had never been called. Every beginner carries the same unspoken assumption: the function returned, so the data is on the disk. And it is flat wrong. The gap between "write() returned" and "the platter holds my bytes" is exactly where real, customer-losing data loss lives. So before we can defend against it, we have to see it plainly. A successful return from write() is a promise the kernel has not yet kept.

So watch what physically happens on that call. write() is a syscall. You cross the border out of your process and into the kernel. (Volume 2's recurring boundary: what crosses is bytes, never live objects.) But the kernel does not turn around and speak to the storage device. It copies your bytes out of your process's memory into pages of the page cache — kernel-owned RAM that mirrors file contents. It flips those pages to dirty (the in-kernel flag is literally PG_dirty), bumps the file's in-memory size, and returns the byte count. That is the entire fast-path syscall: a memcpy plus a little bookkeeping. No seek, no platter, no flash — zero device I/O. Your "saved" record is sitting in volatile RAM that merely happens to belong to the kernel instead of to you.

The dirty pages reach the device later, asynchronously, driven by kernel writeback threads. You may know the old names pdflush/bdflush. Modern kernels use per-backing-device kworker writeback. "Later" is not vague. It is governed by three independent triggers, and knowing them tells you precisely how long your data hangs in the wind. First, a timer. A page that has been dirty longer than vm.dirty_expire_centisecs (about 30 seconds) becomes eligible, and a sweep runs every vm.dirty_writeback_centisecs (about 5 seconds) to flush the eligible ones. Second, memory pressure. Once dirty pages exceed vm.dirty_background_ratio the kernel wakes writeback in the background. If they climb to vm.dirty_ratio, your own next write() is forced to block until pages drain — the one moment write-back stops feeling free. Third, an explicit sync/fsync, which is the whole subject of this chapter.

Write-back is a deliberate trade, not an accident, and it helps to see its two siblings. Write-through caching would push each write to the device before returning. That's durable immediately, but every write() pays a device round-trip, so it is agonizingly slow. O_DIRECT goes the other way and bypasses the page cache entirely, handing I/O straight to the device. Databases use it to manage their own caching. Ordinary buffered I/O, what you get by default, chose write-back on purpose: speed now, durability on demand. The demand is the syscall you'll meet two sections from now.

You can watch the lie with a stopwatch. A hundred writes should touch the disk a hundred times and cost real milliseconds. Unless they don't touch the disk at all:

import time

with open("set.log", "wb") as f:
    start = time.perf_counter()
    for i in range(100):
        n = f.write(b"track played\n")   # returns 13 — bytes copied into cache
    elapsed = time.perf_counter() - start

print(elapsed)     # ~0.0001 s — 100 "writes", not one disk seek

A hundred microseconds for a hundred writes. That number is impossible if the platter were involved. A single seek on spinning rust is milliseconds, tens of thousands of times longer. The speed is the tell: write() returned the byte count the moment it finished the memcpy into cache, and nothing has left RAM. The return value 13 means "13 bytes copied," not "13 bytes persisted." Read that sentence again, because the rest of the chapter is nothing but the machinery for closing the gap between those two claims.

So write() only reaches the kernel's cache. The responsible move is to call f.flush() — but that closes a different gap than you think, because there is a second buffer you didn't know you had, living inside Python itself. →

02Python's buffer sits above the kernel's

You've learned that write() only reaches the page cache, so you reach for f.flush(), feeling diligent. Here is the trap that catches even experienced programmers. They believe flush() means "now it's on disk." It does not. It means "now it's in the kernel." There is a second buffer between your string and the platter, one that lives inside Python, in your own process. And flush() empties only that one. Until you can see both buffers with your own eyes, you cannot reason about how far any given call moves your data. So let's make them visible.

When you open() a file in the normal, buffered mode, Python does not hand you a bare file descriptor. It hands you a tower of wrappers. At the bottom is a FileIO object holding the raw OS file descriptor. Over it sits an io.BufferedWriter — an in-process bytearray, sized to io.DEFAULT_BUFFER_SIZE (8192 bytes, or the filesystem's block size). When you open in text mode there's a third floor on top: a TextIOWrapper that encodes your str into bytes (UTF-8, per Chapter 15) with its own buffering. You can pull the tower apart by hand:

>>> f = open("set.log", "w")
>>> type(f)
<class '_io.TextIOWrapper'>      # floor 3: str -> bytes
>>> type(f.buffer)
<class '_io.BufferedWriter'>     # floor 2: the bytearray in YOUR heap
>>> type(f.buffer.raw)
<class '_io.FileIO'>             # floor 1: the raw OS file descriptor
>>> f.buffer.raw.fileno()
3                            # the integer the kernel knows you by

Now the crucial mechanics. When you call f.write(s) in buffered mode, Python appends the bytes to that BufferedWriter bytearray, and no syscall happens. None. The data has not left your process. The kernel has never heard of it. Only three things push those bytes onward: the bytearray fills to 8192 bytes, or you call f.flush(), or you close(). Any of those makes Python issue the actual write() syscall(s) that push the bytes down into the kernel's page cache. So there are two distinct, stacked buffers between your string and the medium. First, (1) Python's BufferedWriter, in your process's heap. Second, (2) the kernel page cache, in kernel RAM. Two owners, two chunks of volatile memory, one border between them.

And f.flush() operates on exactly one of those borders. It drains Python's bytearray by issuing the write() syscall, pushing the bytes down into the kernel page cache. And it stops there. It issues no fsync. It triggers no device I/O. Nothing reaches the platter. After flush(), your data has simply moved from one volatile buffer to a different volatile buffer. You closed the near gap and left the more dangerous one wide open — the one that survives power loss. You can prove the whole story with os.stat, which reports the file size the kernel knows, not the bytes trapped in Python:

import os

p = "buf.log"
with open(p, "wb") as f:
    f.write(b"x" * 10)
    print(os.stat(p).st_size)   # 0  -> bytes still in Python's bytearray
    f.flush()
    print(os.stat(p).st_size)   # 10 -> flush() crossed into the kernel

Zero, then ten. Between those two prints your bytes physically travelled from your heap to the kernel's, and not one inch further. That jump from 0 to 10 is the Python→kernel boundary being crossed, rendered as an integer you can watch change.

you type
# buffering.py -- who is holding your bytes, and what makes them move
import io, os
from pathlib import Path

P = Path("buf.log")
print("io.DEFAULT_BUFFER_SIZE:", io.DEFAULT_BUFFER_SIZE, "bytes")

# buffering=-1 (the default): block-buffered. Bytes sit in YOUR heap.
with open(P, "wb") as f:
    f.write(b"x" * 10)
    print("block-buffered  after write :", os.stat(P).st_size, "bytes in the kernel")
    f.flush()
    print("block-buffered  after flush :", os.stat(P).st_size, "bytes in the kernel")

# buffering=1: line-buffered. The newline is the trigger. TEXT MODE ONLY.
with open(P, "w", encoding="utf-8", newline="\n", buffering=1) as f:
    f.write("Blinding Lights")
    print("line-buffered   no newline  :", os.stat(P).st_size, "bytes in the kernel")
    f.write("\n")
    print("line-buffered   newline sent:", os.stat(P).st_size, "bytes in the kernel")

# buffering=0: unbuffered. Every write() is its own syscall. BINARY MODE ONLY.
with open(P, "wb", buffering=0) as f:
    f.write(b"Titanium")
    print("unbuffered      after write :", os.stat(P).st_size, "bytes  (no flush called)")

try:
    open(P, "w", encoding="utf-8", buffering=0)
except ValueError as e:
    print("text + buffering=0          :", type(e).__name__ + ":", e)

with open(P, "w", encoding="utf-8") as f:      # the tower, by name
    print("floors                      :", type(f).__name__, "->",
          type(f.buffer).__name__, "->", type(f.buffer.raw).__name__, "| fd", f.fileno())

you see
io.DEFAULT_BUFFER_SIZE: 8192 bytes
block-buffered  after write : 0 bytes in the kernel
block-buffered  after flush : 10 bytes in the kernel
line-buffered   no newline  : 0 bytes in the kernel
line-buffered   newline sent: 16 bytes in the kernel
unbuffered      after write : 8 bytes  (no flush called)
text + buffering=0          : ValueError: can't have unbuffered text I/O
floors                      : TextIOWrapper -> BufferedWriter -> FileIO | fd 3

>>> import os, time
>>> f = open("paid.log", "wb"); f.write(b"order 1041 paid\n")
>>> t = time.perf_counter(); f.flush(); round((time.perf_counter() - t) * 1000, 3)
>>> os.stat("paid.log").st_size, open("paid.log", "rb").read()
>>> t = time.perf_counter(); os.fsync(f.fileno()); round((time.perf_counter() - t) * 1000, 3)
>>> os.stat("paid.log").st_size, open("paid.log", "rb").read()

16
0.071
(16, b'order 1041 paid\n')
6.577
(16, b'order 1041 paid\n')

Two buffers down, and both are volatile RAM. So what actually forces bytes onto the physical medium — past the kernel cache and the drive's own cache? There is exactly one syscall for it. →

# buffer_watch.py -- ask a SECOND process what it can see while the first one holds the bytes
import os, subprocess, sys
from pathlib import Path

LOG = Path("plays.log")
LOG.unlink(missing_ok=True)

READER = (
    "import sys, pathlib;"
    "p = pathlib.Path(sys.argv[1]);"
    "print('        another process reads:', repr(p.read_text(encoding='utf-8')))"
)

# our OWN stdout is buffered too, so we flush=True to keep this transcript in order
def ask_another_process():                       # a brand-new interpreter, its own memory
    subprocess.run([sys.executable, "-c", READER, str(LOG)], check=True)

with open(LOG, "w", encoding="utf-8", newline="\n") as f:
    f.write("Blinding Lights\n")                 # -> Python's BufferedWriter, in THIS heap
    print("after write() : os.stat says", os.stat(LOG).st_size, "bytes", flush=True)
    ask_another_process()

    f.flush()                                    # -> the write() syscall, into the kernel
    print("after flush() : os.stat says", os.stat(LOG).st_size, "bytes", flush=True)
    ask_another_process()

print("after close() : os.stat says", os.stat(LOG).st_size, "bytes")
print("the second process never saw the buffer -- it is not shared memory, it is YOUR heap")

after write() : os.stat says 0 bytes
        another process reads: ''
after flush() : os.stat says 16 bytes
        another process reads: 'Blinding Lights\n'
after close() : os.stat says 16 bytes
the second process never saw the buffer -- it is not shared memory, it is YOUR heap

03fsync is the only real durability guarantee

Two buffers down, one to go — and it's the one that survives the power cut. You now know flush() lands your bytes in the kernel page cache, still volatile. So what actually drives them onto the physical medium? There is precisely one syscall for the job, and if you never call it, no amount of flushing will save your data from a crash. This is the section that hands you the only real durability primitive there is: os.fsync.

os.fsync(fd) tells the kernel: take every dirty page belonging to this file, schedule them to the device now, and do not return control to my process until the storage controller confirms they are physically persisted. That last clause is the whole difference. Where write() and flush() return after a memcpy, fsync performs real device I/O and blocks until it completes. That is why it costs orders of magnitude more: single-digit milliseconds on flash, tens of milliseconds on a spinning disk, versus microseconds for a buffer copy. When fsync returns, and only then, your bytes are on the medium.

There's a subtlety worth stating exactly, because it upgrades the mental model you've been building. On a correctly implemented stack, fsync doesn't only flush the kernel page cache. It also issues a cache-flush / FUA barrier to the drive's own volatile write cache, forcing bytes past that final buffer onto the platter or NAND. So the honest picture has three volatile stages, not two: Python's bytearray, the kernel page cache, and the drive cache. flush() defeats stage 1. fsync() defeats stages 2 and 3 together. This is exactly why serious databases care about drives with "power-loss protection." Some consumer SSDs acknowledge the barrier without honoring it, lying about durability to win benchmarks.

import os

def durable_write(path, data):
    with open(path, "wb") as f:
        f.write(data)          # -> Python's BufferedWriter (your heap)
        f.flush()              # -> kernel page cache (still volatile)
        os.fsync(f.fileno())   # -> the physical medium (durable at last)

Three lines, three floors of the tower, one call per floor. Notice what close() does and does not do. (It's the implicit exit of the with block.) It flushes Python's buffer to the kernel, so closing never strands bytes in your heap. But it does not fsync. A closed file can still be a lost file. Durability is never automatic. You ask for it by name.

you type
# fsync_demo.py -- the only call that reaches the medium, and what it costs
import os, time
from pathlib import Path

P = Path("durable.log")

# ---- the order law: fsync can only push what the kernel already holds ----
with open(P, "wb") as f:
    f.write(b"Blinding Lights\n")
    os.fsync(f.fileno())                    # wrong order: nothing crossed yet
    print("fsync, no flush first :", os.stat(P).st_size, "bytes reached the kernel")
    f.flush()                               # step 1: heap  -> kernel
    os.fsync(f.fileno())                    # step 2: kernel -> medium
    print("flush THEN fsync      :", os.stat(P).st_size, "bytes, now on the medium")

# ---- the price, measured on this machine ----
ROWS = [b"track played\n"] * 500

def bench(rows, durable):
    with open(P, "wb") as f:
        start = time.perf_counter()
        for r in rows:
            f.write(r)
            f.flush()                       # heap -> kernel every time
            if durable:
                os.fsync(f.fileno())        # kernel -> medium every time
        return time.perf_counter() - start

fast = bench(ROWS, durable=False)
slow = bench(ROWS, durable=True)
print(f"500 records, flush only  : {fast * 1000:8.2f} ms")
print(f"500 records, flush+fsync : {slow * 1000:8.2f} ms")
print(f"one barrier costs        : {(slow - fast) / 500 * 1000:8.2f} ms")
print(f"durability is            : {slow / fast:8.0f}x slower, and it is not optional")

you see
fsync, no flush first : 0 bytes reached the kernel
flush THEN fsync      : 16 bytes, now on the medium
500 records, flush only  :     1.43 ms
500 records, flush+fsync :   141.40 ms
one barrier costs        :     0.28 ms
durability is            :       99x slower, and it is not optional

And you can feel its price with a stopwatch. Run ten thousand tiny records two ways: flush only, versus flush-plus-fsync. The gap is not subtle:

import os, time

def bench(use_fsync):
    with open("t.log", "wb") as f:
        start = time.perf_counter()
        for _ in range(10_000):
            f.write(b"row\n")
            f.flush()
            if use_fsync:
                os.fsync(f.fileno())
        return time.perf_counter() - start

print(bench(False))   # ~0.02 s   — all buffer copies
print(bench(True))    # ~20-30 s  — 1000x, each ms a device round-trip

A thousand-fold. That entire difference is device physics. The flush-only run never left RAM, while the fsync run waited on the storage controller ten thousand separate times. This is the wow that reframes the chapter. Durability has a fixed, unavoidable, per-call price measured in device round-trips. And every storage system ever built is, at heart, a scheme for paying that price as few times as possible. (You'll see the scheme in the final section.) If your file's mtime and size don't matter for readback, os.fdatasync is the cheaper cousin. It persists the file data but skips the extra metadata barrier: same data durability, one fewer barrier per call.

Now you hold all three primitives — write, flush, fsync — and each leaves your data in a different buffer. The payoff is being able to point at any line and name exactly what a crash there would destroy. →

# durable_log.py -- chapter 15's append_log, hardened, and the bill for it
import os, time
from datetime import datetime
from pathlib import Path

LOG = Path("plays.log")
PLAYED = ["Blinding Lights", "Titanium"]

def append_plays(path, titles, mode):
    stamp = datetime.now().strftime("%H:%M:%S")
    with open(path, "a", encoding="utf-8", newline="\n") as f:   # 'a' from chapter 15
        for title in titles:
            f.write(f"{stamp}  played  {title}\n")
            if mode == "per-line":
                f.flush()
                os.fsync(f.fileno())          # a barrier for every single line
        f.flush()                             # heap -> kernel, always
        if mode == "per-batch":
            os.fsync(f.fileno())              # one barrier for the whole batch

def bench(mode, batches=200):
    LOG.unlink(missing_ok=True)
    start = time.perf_counter()
    for _ in range(batches):
        append_plays(LOG, PLAYED, mode)
    return (time.perf_counter() - start) / batches * 1000

flush_only = bench("flush-only")
per_batch  = bench("per-batch")
per_line   = bench("per-line")

print(f"flush only          : {flush_only:6.2f} ms per batch   survives a crash, not a power cut")
print(f"fsync per batch     : {per_batch:6.2f} ms per batch   survives both, 1 barrier")
print(f"fsync per line      : {per_line:6.2f} ms per batch   survives both, 2 barriers")
print(f"the barrier costs   : {(per_batch - flush_only):6.2f} ms - and it is per CALL, not per byte")
print("lines in the log    :", len(LOG.read_text(encoding="utf-8").splitlines()))

# ------------- what it prints -------------
flush only          :   0.23 ms per batch   survives a crash, not a power cut
fsync per batch     :   0.73 ms per batch   survives both, 1 barrier
fsync per line      :   0.99 ms per batch   survives both, 2 barriers
the barrier costs   :   0.49 ms - and it is per CALL, not per byte
lines in the log    : 400

04The two crash windows, side by side

You have the three primitives. Here is the payoff: you can look at a specific line of your save routine and say, with certainty, what a crash on that line would destroy. Vague dread — "I might lose data" — is useless. A real durability argument names the exact window and the exact bytes at risk. This section turns your buffer knowledge into a crash-survival map you can read off any code that touches a file. And it hinges on a distinction most programmers never make. A process crash and a power cut are not the same event, and they claim different victims.

The method is simple. Lay a crash on a timeline of your three calls and ask one question: which buffers currently hold the only copy of the data? Because each buffer has a different owner and a different lifetime, the answer changes as your bytes descend the tower. There are two named windows, each defined by which call has not yet run.

Window A — after write(), before flush(). The bytes live only in Python's user-space BufferedWriter, inside your process's heap. The kernel has never seen them. A crash here loses them, even a clean kill -9 with the power cable untouched. A killed process's heap simply evaporates, and nothing flushes a dead process's buffer. Note the asymmetry that makes this window uniquely dangerous: it is lost on a mere process crash, not just power loss. An unhandled exception, a segfault in a C extension, an OOM kill — any of these, in Window A, and the bytes are gone.

Window B — after flush(), before fsync(). Now the bytes are out of Python and in the kernel page cache. Here the asymmetry flips in your favor: a process crash is now survivable. The kernel owns those pages, and it will write them back on its own schedule whether or not your program still exists. So kill -9, an exception, an interpreter crash — none of them lose the data in Window B, because the data no longer depends on your process being alive. But a power cut or a kernel panic in Window B still loses everything, because the page cache is volatile RAM that writeback hadn't yet drained. Only when fsync returns are you clear of both windows: the bytes are on the medium and survive power loss.

You can make the abstract windows concrete. Build a save routine that crashes on command, plus a reader that reports what a fresh process would recover:

import os

def save(f, data, crash_at=None):
    f.write(data)
    if crash_at == "window_A":      # bytes only in Python's heap
        raise SystemExit            # process dies: data LOST
    f.flush()
    if crash_at == "window_B":      # bytes in kernel page cache
        raise SystemExit            # process dies: data SURVIVES
    os.fsync(f.fileno())        # past both windows: durable

Crash in window_A, reopen the file in a brand-new process, and the record is missing. The kernel was never told. Crash in window_B and the same reopen finds the record intact, because the kernel finished the job your dead process couldn't. That single behavioral difference is the two-window model made real: same crash type, opposite outcome, decided entirely by whether flush() had run. To model the power cut you need a simulator, because a real power cut also wipes the page cache. Picture a three-band model where power_cut() clears both python_buf and page_cache but keeps medium. Run it at each stage, and it prints exactly which records survive.

You can now protect a single write. But real programs face a nastier failure — a crash halfway through overwriting a file that was already good — and there fsync alone isn't enough. →

# crash_the_process.py -- kill a real process in each window, then read the file from a live one
import os, subprocess, sys
from pathlib import Path

LOG = Path("windows.log")

CHILD = r'''
import os, sys
stage = sys.argv[1]
with open("windows.log", "wb") as f:
    f.write(b"order 1041 confirmed\n")   # -> Python's BufferedWriter, in the CHILD's heap
    if stage == "A":
        os._exit(1)                      # Window A: die with the bytes still in the heap
    f.flush()                            # -> the kernel page cache
    if stage == "B":
        os._exit(1)                      # Window B: die with the kernel holding them
    os.fsync(f.fileno())                 # -> the medium
    os._exit(1)                          # past both windows
'''

STAGES = [("A", "die after write()  "),
          ("B", "die after flush()  "),
          ("C", "die after fsync()  ")]

for stage, label in STAGES:
    LOG.unlink(missing_ok=True)
    subprocess.run([sys.executable, "-c", CHILD, stage])      # a real process, really killed
    print(label, "-> this process now reads:", repr(LOG.read_bytes()))

print()
print("os._exit() skips every cleanup CPython would do -- no flush, no close, no atexit.")
print("Window B survives because the kernel, not the dead child, owns those pages now.")
print("What no test on one machine can show you: a power cut also empties the kernel's.")

die after write()   -> this process now reads: b''
die after flush()   -> this process now reads: b'order 1041 confirmed\n'
die after fsync()   -> this process now reads: b'order 1041 confirmed\n'

os._exit() skips every cleanup CPython would do -- no flush, no close, no atexit.
Window B survives because the kernel, not the dead child, owns those pages now.
What no test on one machine can show you: a power cut also empties the kernel's.

05Write-temp-then-rename: the atomic save

fsync makes a write durable, but it does nothing about a nastier, more common disaster: a crash halfway through overwriting an existing good file. Picture updating config.json. You open it for writing, and before you've written a single new byte, you have neither the old complete version nor a complete new one. You have a truncated husk, and the good data you were replacing is gone forever. Every config editor, every save-game, every document app must guarantee one invariant: a reader always sees SOME valid file, never a blend of old and new. This section teaches the filesystem trick that makes a half-written file impossible to observe: the atomic rename.

First, see the trap precisely. open(target, "wb") truncates the existing file to zero length as part of opening it — before your first write(). So the destruction happens up front, and the window of vulnerability is the entire duration of your write. A crash anywhere in it leaves target corrupt, and the old contents are already destroyed. You can reproduce it in two lines: open for writing, raise an exception before writing anything, and os.stat(target).st_size is now 0. The good file is gone and you never wrote the new one.

The fix exploits a POSIX guarantee that is one of the quiet load-bearing facts of all of computing. rename(2), which is Python's os.replace, is atomic within a single filesystem. A rename doesn't move bytes. It flips a directory entry from pointing at the old inode to pointing at the new inode, in one indivisible metadata operation. No observer, and no crash, can catch it half-done. Any reader that looks sees either the complete old file or the complete new file, never a partial one. So instead of editing the target in place, you build the new version off to the side. Then you swing the name over to it in a single atomic instant.

import os, tempfile

def atomic_write(path, data):
    d = os.path.dirname(path) or "."
    fd, tmp = tempfile.mkstemp(dir=d)        # same dir -> same filesystem
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())            # step 2: temp's DATA is durable
        os.replace(tmp, path)                # step 3: atomic name flip
        dirfd = os.open(d, os.O_RDONLY)      # step 4: persist the rename itself
        try:
            os.fsync(dirfd)
        finally:
            os.close(dirfd)
    except BaseException:
        os.unlink(tmp)                        # never leave a stray temp behind
        raise

Walk the failure modes, because each line is defending one. If you skip step 1 and write in place, you already saw the truncated-husk disaster. If you put the temp in /tmp instead of the target's directory, you may cross a filesystem boundary, since /tmp is often a separate mount. Then os.replace silently degrades from an atomic rename into a non-atomic copy-then-delete, reopening exactly the window you were trying to close. If you skip step 2, you can atomically rename an empty or partial temp file over your good one. A crash leaves target pointing at durable garbage: durability without the data, the worst of both worlds. And if you skip step 4, the rename is durable-in-name-only. The directory entry change sat in the page cache, and a power cut right after replace can un-happen the swap even though the temp file's data was safely persisted. Do all four and the invariant holds through any crash: target always names a complete, fsync'd file.

you type
# atomic.py -- the four-step ritual, and the disaster it exists to prevent
import os, tempfile
from pathlib import Path

TARGET = Path("config.json")
GOOD = b'{"volume": 7, "shuffle": true}\n'
NEW  = b'{"volume": 9, "shuffle": false}\n'

def fsync_dir(d):                                  # step 4 -- POSIX only, no-op on Windows
    if os.name == "nt":
        return
    fd = os.open(d, os.O_RDONLY)
    try:
        os.fsync(fd)
    finally:
        os.close(fd)

def atomic_write(path, data, crash=False):
    d = os.path.dirname(os.path.abspath(path))
    fd, tmp = tempfile.mkstemp(dir=d)              # step 1: same dir -> same filesystem
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(data)
            if crash:
                raise RuntimeError("crash, mid-write")
            f.flush()
            os.fsync(f.fileno())                   # step 2: the temp file's data is durable
        os.replace(tmp, path)                      # step 3: one atomic name flip
        fsync_dir(d)                               # step 4: persist the rename itself
    except BaseException:
        os.unlink(tmp)                             # never leave a stray temp behind
        raise

# --- the disaster: overwrite in place, crash before writing one byte ---
TARGET.write_bytes(GOOD)
try:
    with open(TARGET, "wb") as f:                  # "wb" truncates AT OPEN TIME
        raise RuntimeError("crash, mid-write")
except RuntimeError:
    pass
print("in place, crashed  :", os.stat(TARGET).st_size, "bytes ->", TARGET.read_bytes())

# --- the ritual: the same crash, at the same moment ---
TARGET.write_bytes(GOOD)
try:
    atomic_write(TARGET, NEW, crash=True)
except RuntimeError:
    pass
print("atomic,   crashed  :", os.stat(TARGET).st_size, "bytes ->", TARGET.read_bytes())
print("temp files left    :", sorted(p.name for p in Path(".").glob("tmp*")))

atomic_write(TARGET, NEW)                          # and now let it finish
print("atomic,   completed:", os.stat(TARGET).st_size, "bytes ->", TARGET.read_bytes())

you see
in place, crashed  : 0 bytes -> b''
atomic,   crashed  : 31 bytes -> b'{"volume": 7, "shuffle": true}\n'
temp files left    : []
atomic,   completed: 32 bytes -> b'{"volume": 9, "shuffle": false}\n'

Now you can save one file durably and atomically. But do it in a loop — one fsync per record — and your program crawls. Databases commit tens of thousands of durable writes a second on the same disk. Here's the trick. →

# durable_save.py -- chapter 15's playlist save, hardened: flush, fsync, atomic replace
import os, subprocess, sys, tempfile
from pathlib import Path

PLAYLIST = Path("playlist.txt")
TRACKS = ["Blinding Lights", "Titanium", "Levels", "bad guy"]

def fsync_dir(d):                                    # step 4 -- POSIX only
    if os.name == "nt":                              # Windows cannot open a directory
        return
    fd = os.open(d, os.O_RDONLY)
    try:
        os.fsync(fd)
    finally:
        os.close(fd)

def durable_save(path, tracks, crash=False):
    d = os.path.dirname(os.path.abspath(path))
    fd, tmp = tempfile.mkstemp(dir=d, prefix=".playlist-", suffix=".tmp")   # step 1
    try:
        with os.fdopen(fd, "w", encoding="utf-8", newline="\n") as f:
            for title in tracks:
                f.write(title + "\n")
            f.flush()                                # heap   -> kernel
            os.fsync(f.fileno())                     # step 2: kernel -> medium
        if crash:
            os._exit(1)                              # killed between fsync and replace
        os.replace(tmp, path)                        # step 3: the atomic name flip
        fsync_dir(d)                                 # step 4: persist the rename itself
    except BaseException:
        os.unlink(tmp)
        raise

def load(path):
    return path.read_text(encoding="utf-8").splitlines() if path.exists() else []

if __name__ == "__main__":
    if "--crash" in sys.argv:                        # the doomed child process
        durable_save(PLAYLIST, ["JUNK", "HALF WRITTEN"], crash=True)

    durable_save(PLAYLIST, TRACKS)
    print("saved            :", load(PLAYLIST))

    subprocess.run([sys.executable, __file__, "--crash"])     # a real process, really killed
    print("after that crash :", load(PLAYLIST))
    strays = sorted(Path(".").glob(".playlist-*.tmp"))
    print("stray temp files :", len(strays), "- the target was never in danger, but litter is real")
    for s in strays:
        s.unlink()                                   # sweep at startup, every real program does

    durable_save(PLAYLIST, load(PLAYLIST) + ["Don't Start Now"])
    print("saved            :", load(PLAYLIST))

saved            : ['Blinding Lights', 'Titanium', 'Levels', 'bad guy']
after that crash : ['Blinding Lights', 'Titanium', 'Levels', 'bad guy']
stray temp files : 1 - the target was never in danger, but litter is real
saved            : ['Blinding Lights', 'Titanium', 'Levels', 'bad guy', "Don't Start Now"]

06Durability costs throughput — how databases cheat

You now know the honest, safe save: fsync every record. Put it in a loop and watch your program crawl. You get a few hundred durable writes per second, because each fsync blocks on slow physical I/O and you're paying that price once per record. Yet real databases commit tens of thousands of durable transactions per second on the very same hardware. They are not cheating on durability. Every one of those transactions genuinely survives a power cut. They are amortizing durability's cost with one specific, beautiful trick. It is the design pattern this whole chapter has been building toward: the write-ahead log.

Start from the physics you measured two sections ago. fsync's guarantee comes from waiting on the device, and that wait is a fixed cost per call. It's roughly one device round-trip plus a cache-flush barrier, almost independent of how many bytes you're syncing. Persisting one record and persisting a thousand records in a single fsync cost nearly the same, because the barrier dominates and the extra bytes ride along for free. Read that again, because it's the entire insight: fsync's cost is per-call, not per-byte. If you call it once per record, throughput is capped at about 1/latency. That's a few hundred to a few thousand ops per second, forever, no matter how fast your CPU. The only way up is to make one fsync do the work of many.

A write-ahead log does exactly that, by combining two moves. First, instead of updating scattered data structures in place — random I/O, an fsync for each — you append every change as a record to the end of a single log file. Appends are sequential writes: no seeks, cache-friendly, the fastest thing a storage device does. Second, you fsync the log once for a whole batch of appended records, making them all durable together with a single barrier. Many small durable writes collapse into one. Here are the two strategies side by side, and the second is the entire ballgame:

import os

def commit_each(records):        # one barrier PER record — slow
    with open("wal.log", "ab") as f:
        for r in records:
            f.write(r + b"\n")
            f.flush()
            os.fsync(f.fileno())   # N records -> N device round-trips

def commit_group(records):       # ONE barrier for the whole batch — fast
    with open("wal.log", "ab") as f:
        for r in records:
            f.write(r + b"\n")   # sequential appends, all in cache
        f.flush()
        os.fsync(f.fileno())       # N records -> 1 device round-trip

Feed both a thousand records. commit_each pays a thousand barriers, commit_group pays one, and the throughput gap is roughly a thousand-fold. It's the same ratio you measured in section 3, now working for you instead of against you. This is group commit, and a real database doesn't even need the records to arrive in one call. It accumulates the commit records of many concurrent transactions arriving within a small time window (a millisecond or two). It fsyncs once, then acknowledges them all together. Each transaction pays a tiny added latency, waiting a moment for the group to form. In exchange it gets a massive throughput gain and full durability. Latency up a hair, throughput up 100×, durability uncompromised.

def replay(path):                  # a database's crash recovery, in miniature
    state = {}
    with open(path, "rb") as f:
        for line in f:
            key, val = line.rstrip(b"\n").split(b"=", 1)
            state[key] = val          # redo every durable change, in order
    return state

That is the core of a storage engine in about twenty lines: append changes to a durable log, replay it to rebuild state after a crash. Everything else — B-trees, MVCC, checkpoints that truncate the log once the data files catch up — is optimization on top of this spine. And notice the reframe of the whole chapter. fsync is the expensive, irreducible primitive, and the write-ahead log is the design pattern that makes durability affordable. It is literally the temp-file and fsync ideas of the last two sections, scaled from one file to a production database: write it down safely first, make the durable copy the source of truth. You didn't learn six disconnected tricks. You learned one law (durability costs a device round-trip) and the single pattern the entire industry uses to pay it as rarely as possible.

Traceback (most recent call last):
  File "append_log.py", line 8, in <module>
    append_play("plays.log", "Blinding Lights")
  File "append_log.py", line 6, in append_play
    os.fsync(f.fileno())            # now make it survive a power cut
             ^^^^^^^^^^
ValueError: I/O operation on closed file

Traceback (most recent call last):
  File "save_config.py", line 6, in <module>
    os.fsync(fd)                        # step 2 of the ritual, one line too late
    ^^^^^^^^^^^^
OSError: [Errno 9] Bad file descriptor

Traceback (most recent call last):
  File "test_durable.py", line 8, in <module>
    durable_write(io.BytesIO(), b"order 1041 paid\n")   # a stand-in for the real file
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "test_durable.py", line 6, in durable_write
    os.fsync(stream.fileno())
             ^^^^^^^^^^^^^^^
io.UnsupportedOperation: fileno

# when_not_to_fsync.py -- the policy is a decision about what losing the tail costs
#
# (a) the derived search index -- NO fsync. A power cut costs a 40-second rebuild
#     from data that is already durable. Paying 0.3 ms per record to protect a
#     copy you can regenerate is buying insurance on a photocopy.
# (b) the telemetry stream -- fsync on a TIMER, not per record. The dashboard
#     tolerates a two-second hole; 50,000 barriers a second is not survivable.
# (c) the row that says a customer paid -- fsync, every time, before you answer
#     "paid". There is no rebuild. The only copy of that fact is the one you keep.
import os, time
from pathlib import Path

P = Path("policy.log")
N = 2000

def run(policy, every=0):
    P.unlink(missing_ok=True)
    with open(P, "wb") as f:
        start = time.perf_counter()
        for i in range(N):
            f.write(b"cpu=0.41 mem=0.68\n")
            if policy == "per-record":
                f.flush()
                os.fsync(f.fileno())
            elif policy == "every-100" and i % 100 == 99:
                f.flush()
                os.fsync(f.fileno())
        f.flush()
        if policy != "no-fsync":
            os.fsync(f.fileno())
        return time.perf_counter() - start

PLAN = [("no-fsync",   0,   "the whole tail, up to 5 seconds of it"),
        ("every-100",  100, "at most 99 records"),
        ("per-record", 0,   "nothing at all")]

for policy, every, risk in PLAN:
    secs = run(policy, every)
    print(f"{policy:11} {N / secs:10,.0f} records/sec   power cut costs: {risk}")

# ------------- what it prints -------------
no-fsync     5,885,815 records/sec   power cut costs: the whole tail, up to 5 seconds of it
every-100      275,843 records/sec   power cut costs: at most 99 records
per-record       3,430 records/sec   power cut costs: nothing at all
#
# One laptop, one NVMe SSD, one run - your digits will differ and the shape will not.
# Between the top row and the bottom row sits a factor of about 1,700, bought with
# nothing but barriers. That is the whole negotiation: throughput on one side,
# the width of the crash window on the other, and no policy that wins both.

You've flattened objects to bytes and forced those bytes durably onto disk. Next: what happens when the bytes have to cross to another process — where even the object's shape has to travel. Serialization and the pickle pipe. →