30The capstone — ship a real app

In Chapter 29 we took a slow program, ran a profiler over it, and made the one line that actually mattered fast. In this one we do the hardest thing in the whole volume. We take everything and ship it. Here's the plan. The two songs that have travelled with us since Chapter 0 — "Blinding Lights" at 200 seconds and "Titanium" at 245 — finally get a real home. That home is a playlist manager a stranger can install, start, trust, and change, built from nothing but the standard library. Every skill you proved once, alone, in a scratch file — the dataclass, the atomic save, the profiler pass — now has to meet the others in one running program. And the whole way through we keep asking the one question this volume has circled from the start: what actually crosses the boundary between your program and the disk, the kernel, the next process — the live object, or bytes? By the end we've shipped it. Six clean crossings — module, type, kernel, environment, correctness, performance — meet in a single app that survives being closed. Not one big skill. The sum of six small ones.

01Assemble the pieces

Let's start with the blank file. Open a brand-new folder, create main.py, and look at the cursor blinking at you. That cursor is asking a question no scratch file ever asked. You've spent this whole volume collecting good ideas: an atomic save from chapter 18, a dataclass from chapter 12, a profiler run from chapter 29. Every one of them worked, once, in a scratch file you've since deleted. Now they have to live together, and the cursor wants to know which line goes first? That's the real terror of the blank main.py. It's not that you're short on ideas. It's that you've got twelve of them and no shape to hang them on. Type them all into one file, top to bottom, and by this afternoon you own an 800-line function that runs but can never be read. Change the save format and you're re-reading the shuffle logic just to be sure you didn't break it.

The cure isn't discipline or good taste. Watch what it actually is: a mechanical property of how Python loads code. Once you see it, you can lean the whole app against it. "Separation of concerns" sounds like a slogan. Cash it out and it means something concrete: a unit you can import and exercise without dragging its collaborators into existence. Our app splits into exactly four files. The split is decided by one rule: arrows point down the dependency graph, never up, never in a circle.

app/
  __init__.py     # marks the folder as a package (can be empty)
  model.py        # pure data: Song, Playlist — imports NOTHING of ours
  engine.py       # operations on data: add, remove, shuffle, next_song
  storage.py      # bytes to objects and back — the ONLY file that touches disk
  __main__.py     # the command loop — the ONLY file that touches stdin/stdout

model sits at the bottom because it depends on nothing of ours. It is pure data, constructible in a test with no filesystem, no clock, no I/O. engine imports model. storage imports model. __main__ imports engine and storage. Every seam between them is a plain function signature, like add_song(playlist, song) or save_atomic(path, playlist). And every signature is something a test can call in one line. That is the whole payoff of the layout. Each idea gets placed exactly once, and you can reason about it without holding the other eleven in your head.

To trust this you have to know what an import actually does. The module boundary is not a folder convention. It is a real runtime boundary. A module is an object. The first time import app.model runs, CPython looks for the key 'app.model' in sys.modules, a plain dict that caches every module ever loaded. Not finding it, CPython locates the file and builds a new empty module object. Then it does something crucial: it inserts that object into sys.modules immediately, before running a single line of the body. Only then does it execute model.py top to bottom to populate it. The names you define land in model.__dict__. Every later import app.model, from anywhere, finds the key and hands back that exact same cached object. The file is never run twice.

Now watch why an up-arrow is not a style problem but a correctness bug. Suppose storage.py imports engine while engine.py imports storage. That's a cycle. Python starts importing engine, drops a half-built engine into sys.modules, runs its body, hits from app.storage import save, and dives into storage. Now storage's body hits from app.engine import next_song and finds engine already in the cache. So it does not re-import. It just reads next_song out of engine's namespace. But engine's body never reached the line that defines next_song. It is paused two frames up, mid-import. The name does not exist yet, so Python raises ImportError: cannot import name 'next_song'. That half-initialized module in sys.modules is the whole disease.

The bottom of the graph is model.py — pure data, depended on by everyone, depending on no one. So it had better be exactly right. Next: how to state the shape of a Song once and let the language manufacture the rest. →

02The model: dataclasses with checked hints

You are about to write the same three chunks of boilerplate a hundred times. An __init__ that does nothing but self.title = title; self.artist = artist; self.duration_s = duration_s. An __eq__ so two identical songs compare equal. A __repr__ so that when a test fails it prints something you can actually read instead of <Song object at 0x1f3a>. Hand-writing this is where silent bugs breed. Six months from now you add a tags field, update __init__, and forget to update __eq__. Now two songs that differ only in their tags test as equal, and a deduplication somewhere quietly eats one of them. What you want instead is to state the shape of a Song exactly once. That single statement becomes the source of truth that both the running program and the type checker read.

from dataclasses import dataclass, field

@dataclass(frozen=True)
class Song:
    title: str
    artist: str
    duration_s: int
    tags: tuple = ()

@dataclass
class Playlist:
    name: str
    songs: list = field(default_factory=list)

Those thirteen lines look declarative, but @dataclass is doing something you should never take on faith: it is writing code. A decorator is just a function. It receives the class object after the class body has finished executing, and it returns a class. dataclass reaches into the class's __annotations__. That's the ordered dict {'title': str, 'artist': str, 'duration_s': int, 'tags': tuple} that Python builds automatically from every name: type line in the body. From that dict it generates source text for __init__, __repr__, and __eq__. Then it compiles that text with exec and attaches the resulting functions to the class. You can see the evidence it leaves behind:

>>> from app.model import Song
>>> Song.__init__.__doc__            # the signature dataclass synthesized
'Song(title: str, artist: str, duration_s: int, tags: tuple = ()) -> None'
>>> list(Song.__dataclass_fields__)   # the Field objects it parsed from the hints
['title', 'artist', 'duration_s', 'tags']
>>> Song('Blinding Lights', 'The Weeknd', 200)
Song(title='Blinding Lights', artist='The Weeknd', duration_s=200, tags=())

Now the sharpest point in the whole chapter, and the one most people never internalize: those type hints have two different audiences who treat them completely differently. At runtime, the annotations are inert. They are values sitting in a dict, and Python never checks them. Song(title=42, artist='x', duration_s=200) runs perfectly and stores the integer 42 where a str was promised. The enforcement is a separate, earlier pass. A static checker — mypy or pyright — reads those very same annotations before the program ever runs and flags the mismatch. So the field hints do double duty. They drive code generation, and they are the contract the checker verifies. The dict feeds both readers, but only one of them cares.

Two more knobs, both earned. frozen=True makes Song generate a __setattr__ that raises FrozenInstanceError on any assignment, plus a __hash__. So a Song becomes a genuine value: immutable, hashable, safe to drop in a set or use as a dict key. That is exactly what you want for the thing that flows through your storage layer and back. And field order matters. A field with no default cannot follow one that has a default, because the generated __init__(self, a, b=1, c) would be an illegal signature. Python raises TypeError at class-definition time, before you ever construct anything.

The model is exact and value-typed. But it lives in RAM, and RAM is volatile — chapter 0's oldest warning. To outlive the process it has to become bytes on a disk, durably, without a crash ever leaving a smear. Next: the save that has only two outcomes. →

03Storage: durable, atomic, serialized

Here is the scenario that makes this section matter more than any other. The user adds "Titanium" to a fifty-song playlist and hits save. Half a second later the battery dies, or the kernel panics, or someone trips over the power cord. They reboot and open the app. What do they see? With the obvious code — open(path, 'w') then json.dump — the answer is often a corrupt, truncated JSON file that crashes the loader. Here's why: open(path, 'w') truncates the real file to zero bytes the instant it opens, then streams the new bytes in. A crash anywhere in that window leaves a half-written smear where fifty good songs used to be. You did not lose the one edit. You lost everything. What you need is a save with exactly two possible outcomes from the reader's view: the complete old playlist or the complete new one. And the data must be genuinely on the disk platter, not merely promised.

That is two independent guarantees, and you need both. The first is atomicity via rename. On POSIX, rename(2) over an existing path is atomic: the directory entry is repointed from the old inode to the new one in a single indivisible operation. Any reader who opens the target sees one inode or the other, never a blend. So the recipe is: write the entire new content to a temporary file in the same directory, then rename the temp over the target. Same-directory is not optional — a rename across filesystems is secretly a copy-then-delete, which is not atomic, and Python's os.replace silently falls back to that if the temp is elsewhere. Use os.replace, not os.rename: on Windows os.rename raises if the target exists, while os.replace mimics POSIX overwrite semantics on every platform. That one choice is the cross-platform contract.

The second guarantee is durability via fsync. This is the one people skip, and it is the one the power cut punishes. A plain write() only copies your bytes into the kernel's page cache. They are in RAM, and the disk may not physically see them for seconds. Crash before that writeback and the data is gone, even though write() returned success. os.fsync(fd) blocks until the storage device confirms the bytes are on stable media. Here's the subtle part almost everyone misses. You must fsync the file so the data blocks are durable. You must also fsync the containing directory so the rename itself — the new directory entry — is durable. Skip the directory fsync and a crash can leave your file's data perfectly safe while the directory still points at the old name.

import os, json, tempfile
from dataclasses import asdict
from app.model import Song, Playlist

def save_atomic(path, playlist):
    data = json.dumps(asdict(playlist), indent=2).encode("utf-8")
    folder = os.path.dirname(os.path.abspath(path))
    fd, tmp = tempfile.mkstemp(dir=folder, suffix=".tmp")  # SAME dir
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(data)
            f.flush()               # Python buffer -> kernel page cache
            os.fsync(f.fileno())    # kernel -> physical disk
        os.replace(tmp, path)       # atomic pointer flip, one tick
        dfd = os.open(folder, os.O_RDONLY)
        try:
            os.fsync(dfd)           # make the rename itself durable (POSIX)
        finally:
            os.close(dfd)
    except BaseException:
        os.unlink(tmp)              # never leave a stray temp on failure
        raise

Underneath sits the serialization layer. It exists because json.dump cannot take a Song — JSON has no concept of your types. So you bridge through the dict. asdict(playlist) flattens the whole dataclass tree into nested dicts and lists that JSON understands. This is the volume's law in miniature: the live object is flattened to a byte stream on the way out. On load you must reverse it, because JSON round-trips a Song back into a plain dict. Something has to turn that dict back into a Song and re-validate it:

def load(path):
    with open(path, "rb") as f:
        raw = json.loads(f.read())          # dicts and lists, no Songs yet
    songs = [Song(**{**s, "tags": tuple(s["tags"])}) for s in raw["songs"]]
    return Playlist(name=raw["name"], songs=songs)  # rebuilt + revalidated

The app now saves the way a database does. But a program a stranger cannot start might as well not exist. Next: how Python finds and runs your package, and how to hand someone an environment they can rebuild byte for byte. →

04Structure and environment

"It works on my machine." That sentence is where most projects die, and it usually dies three different ways. A stranger clones your repo and gets ModuleNotFoundError: No module named 'app', because your imports quietly assumed the current directory. Or it starts but misbehaves, because their installed library versions drifted from the ones you developed against and never wrote down. Or, most banal and most fatal, they simply cannot figure out how to start it. What you need here is a program someone who has never spoken to you can reproduce and launch: one unambiguous start command, and an environment they can recreate exactly. "Works on my machine" has to become "works on every machine," and both halves have a concrete mechanism.

Start with running. A package is a directory containing __init__.py. The command python -m app tells the interpreter to import the package named app, then execute its __main__.py as the entry point. Why does -m beat python app/__main__.py? Because of what each does to sys.path[0], the first place Python looks for imports. With -m, Python sets sys.path[0] to the current working directory. So app is on the path, and an absolute intra-package import like from app.engine import next_song resolves cleanly. It also sets __package__, so relative imports work too. Run the loose script instead and sys.path[0] becomes the script's own folder, app/. That puts engine.py on the path but not the app package itself, so from app.engine fails with ModuleNotFoundError. The -m convention is not ceremony. It is the concrete reason your imports resolve.

you type
# ---------- clip.py ----------
"""clip.py -- argparse, the whole shape, on one small command."""
import argparse, sys

p = argparse.ArgumentParser(prog="clip", description="trim one track")
p.add_argument("title", help="the track to trim")            # POSITIONAL: required, by order
p.add_argument("-s", "--seconds", type=int, default=30,      # OPTION: type= runs int() for you
               help="how long to keep (default: %(default)s)")
p.add_argument("--fade", action="store_true", help="fade the last second out")
args = p.parse_args()                    # reads sys.argv[1:] unless you hand it a list

print("sys.argv   =", sys.argv)          # ch 19's raw list, still exactly where it was
print("args       =", args)              # a Namespace: one attribute per name you declared
print(f"trimming {args.title!r} to {args.seconds}s"
      f"{' with a fade' if args.fade else ''}")


# ---------- the terminal ----------
$ python clip.py -h
$ python clip.py Titanium
$ python clip.py Titanium --seconds 45 --fade
$ python clip.py Titanium -s forty

you see
$ python clip.py -h
usage: clip [-h] [-s SECONDS] [--fade] title

trim one track

positional arguments:
  title                 the track to trim

options:
  -h, --help            show this help message and exit
  -s SECONDS, --seconds SECONDS
                        how long to keep (default: 30)
  --fade                fade the last second out

$ python clip.py Titanium
sys.argv   = ['clip.py', 'Titanium']
args       = Namespace(title='Titanium', seconds=30, fade=False)
trimming 'Titanium' to 30s

$ python clip.py Titanium --seconds 45 --fade
sys.argv   = ['clip.py', 'Titanium', '--seconds', '45', '--fade']
args       = Namespace(title='Titanium', seconds=45, fade=True)
trimming 'Titanium' to 45s with a fade

$ python clip.py Titanium -s forty
usage: clip [-h] [-s SECONDS] [--fade] title
clip: error: argument -s/--seconds: invalid int value: 'forty'
   (exit code 2)

import sys
from app.engine import add_song, next_song
from app.storage import load, save_atomic

def main(argv):
    cmd = argv[1] if len(argv) > 1 else "help"
    # ... dispatch cmd to the right engine function ...
    return 0

if __name__ == "__main__":      # true only when launched as the entry point
    sys.exit(main(sys.argv))

Prove the difference to yourself rather than believing it. Print the three values that decide import resolution under each launch:

import sys
print("sys.path[0] =", sys.path[0])
print("__package__ =", __package__)
print("__name__    =", __name__)

# python -m app     -> sys.path[0] = CWD,     __package__ = 'app'   -> imports work
# python app/__main__.py -> sys.path[0] = app/, __package__ = None    -> 'from app...' FAILS

Now the environment. A virtual environment is not magic and not a sandbox. It is a directory with its own python binary (or a symlink to one), its own site-packages, and a small text file, pyvenv.cfg, that points back at the base interpreter that created it. "Activating" it merely prepends the venv's bin (or Scripts) folder to your PATH, so the word python resolves there. The real switch is that this python computes sys.prefix from its own location, so it searches the venv's site-packages first. Isolation is a path trick, fully explainable and fully inspectable:

import sys
print(sys.prefix)         # INSIDE a venv: points at the venv folder
print(sys.base_prefix)    # always the base interpreter that built it
# prefix != base_prefix  =>  you are in a venv. Equal  =>  you are not.

Now anyone can install it and start it. But can anyone trust it? Belief that the code is correct does not survive the next edit. Next: turning "I think it works" into a green suite you can run in one command — and a method for the day it goes red. →

05Proof: tests and a debugging pass

You believe the app is correct. Belief is not evidence, and worse, it does not survive the next change. You tune the shuffle to feel more random, and without noticing you shift an index by one, so next_song now skips the last track. Nothing errors. The suite is in your head, so it never runs. A user finds it three weeks later. What you need is twofold. First, a mechanical proof, runnable in one command, that the claims that matter still hold after every edit. Second, a repeatable method for the moment a test does go red, so "it's broken" becomes "the fault is on line 34, here is the exact bad value, and here is the test that will scream if it ever comes back." A green suite is the difference between a program you hope works and one you can prove works.

from app.model import Song, Playlist
from app.engine import add_song, next_song
from app.storage import save_atomic, load

def test_song_equality_is_by_value():        # model layer — pure, no I/O
    a = Song("Titanium", "Sia", 245)
    b = Song("Titanium", "Sia", 245)
    assert a == b and hash(a) == hash(b)

def test_next_wraps_to_start():                # engine layer — in-memory, no disk
    pl = Playlist("set", [Song("A", "x", 1), Song("B", "y", 2)])
    assert next_song(pl, current=1) == pl.songs[0]  # after last -> first

def test_save_load_round_trips(tmp_path):     # storage layer — tmp_path fixture
    pl = Playlist("set", [Song("Titanium", "Sia", 245)])
    p = tmp_path / "pl.json"
    save_atomic(str(p), pl)
    assert load(str(p)) == pl           # bytes out, bytes in, same object

pytest is three mechanisms stacked: collection, execution, introspection. Collection: it imports every test_*.py, finds functions named test_*, and builds a list of test items. Execution: it calls each one inside its own try/except. An uncaught exception — the AssertionError that a failed assert raises — marks that item failed and moves on. The third mechanism is the one worth opening, because it looks like sorcery. Take a plain assert a == b in ordinary Python. When it fails, it gives you nothing but the bare word AssertionError, no values. Yet pytest prints assert 3 == 4, with the actual operands. How?

Because pytest rewrites the bytecode of your assert statements at import time. It installs an import hook, tapping the very sys.modules loading machinery from section 1. That hook intercepts each test module as it is compiled and walks its AST. It recompiles every assert a == b into instructions that evaluate the subexpressions, stash their values, and format a rich message if the assertion is false. So pytest is quite literally editing your test's syntax tree to make failures legible. It is not a magic assert keyword. It is Python's own import system, bent to a purpose. That is exactly why the same assert run under plain python stays mute.

Notice tmp_path appearing as a parameter with no obvious source. That is a fixture, and fixtures are dependency injection by parameter name. Pytest sees an argument called tmp_path, finds the built-in fixture of that name, calls it, and passes its return value in. Here that value is a fresh per-test temporary directory that never touches your real data. A fixture that yields can run teardown after the test. The three tests above are deliberately layered. Model tests are pure equality on frozen values. Engine tests run on an in-memory playlist. Storage tests round-trip through a throwaway directory. Each layer is testable precisely because section 1 kept the import graph acyclic.

# ---------- the tree: make these folders and files, exactly ----------
medialib-project/
    pyproject.toml
    medialib/                 <- THE PACKAGE. this folder is what ships.
        __init__.py
        tracks.py             <- the model. imports nothing of ours.
        storage.py            <- the ONLY file that touches the disk
        cli.py                <- the ONLY file that reads argv or writes stdout
        __main__.py           <- the front door:  python -m medialib
    tests/
        test_medialib.py      <- OUTSIDE the package, like a stranger


# ---------- medialib/__init__.py ----------
"""medialib -- a tiny media library you can actually ship."""
__version__ = "1.0.0"


# ---------- medialib/tracks.py ----------
"""tracks.py -- the model. Pure data: it imports nothing of ours."""
from dataclasses import dataclass

CATALOGUE = {                        # stands in for the API a downloader would call
    "blinding-lights": ("Blinding Lights", "The Weeknd", 200),
    "titanium": ("Titanium", "David Guetta", 245),
    "levels": ("Levels", "Avicii", 203),
}


@dataclass(frozen=True)              # ch 23: the hints ARE the fields
class Track:
    title: str
    artist: str
    duration_s: int
    source: str = "local"

    @property
    def clock(self) -> str:
        m, s = divmod(self.duration_s, 60)
        return f"{m}:{s:02d}"        # ch 17: the format spec, after the colon


def matches(t: Track, needle: str) -> bool:
    n = needle.casefold()            # ch 17: casefold is for MATCHING
    return n in t.title.casefold() or n in t.artist.casefold()


def from_url(url: str) -> Track:
    """SIMULATED metadata fetch. No network: we look the slug up in CATALOGUE."""
    slug = url.rstrip("/").rsplit("/", 1)[-1]
    title, artist, secs = CATALOGUE[slug]
    return Track(title, artist, secs, source="web")


# ---------- medialib/storage.py ----------
"""storage.py -- the ONLY file that touches the disk."""
import csv, json, os, tempfile
from dataclasses import asdict
from medialib.tracks import Track


def load(path):
    if not os.path.exists(path):
        return []                                    # first run: an empty library
    with open(path, "r", encoding="utf-8") as f:     # ch 15: name the encoding
        return [Track(**row) for row in json.load(f)]


def save_atomic(path, tracks):
    data = json.dumps([asdict(t) for t in tracks], indent=2).encode("utf-8")
    folder = os.path.dirname(os.path.abspath(path))
    fd, tmp = tempfile.mkstemp(dir=folder, suffix=".tmp")   # 1. SAME directory
    try:
        with os.fdopen(fd, "wb") as f:
            f.write(data)
            f.flush()                                # 2. python -> kernel
            os.fsync(f.fileno())                     #    kernel -> the platter
        os.replace(tmp, path)                        # 3. one atomic name flip
    except BaseException:
        os.unlink(tmp)                               # 4. leave no litter
        raise


def export_csv(path, tracks):
    with open(path, "w", newline="", encoding="utf-8") as f:    # newline="" ALWAYS
        w = csv.DictWriter(f, fieldnames=["title", "artist", "duration_s", "source"])
        w.writeheader()
        w.writerows(asdict(t) for t in tracks)
    return len(tracks)


# ---------- medialib/cli.py  (1 of 2): the parser ----------
"""cli.py -- argparse and printing. The only file that reads argv or writes stdout."""
import argparse, logging
from medialib import storage
from medialib.tracks import Track, matches, from_url

log = logging.getLogger("medialib")


def build_parser():
    p = argparse.ArgumentParser(prog="medialib", description="a tiny media library")
    p.add_argument("--db", default="library.json", help="where the library lives")
    p.add_argument("-v", "--verbose", action="store_true", help="turn on DEBUG logging")
    sub = p.add_subparsers(dest="cmd", required=True, metavar="COMMAND")

    a = sub.add_parser("add", help="add one track by hand")
    a.add_argument("title")
    a.add_argument("artist")
    a.add_argument("duration_s", type=int)           # type= runs int() FOR you

    sub.add_parser("list", help="print the whole library")

    s = sub.add_parser("search", help="find tracks by title or artist")
    s.add_argument("needle")

    e = sub.add_parser("export", help="write the library out as CSV")
    e.add_argument("-o", "--out", default="library.csv")

    f = sub.add_parser("fetch", help="SIMULATED metadata fetch -- no network")
    f.add_argument("url")
    return p


# ---------- medialib/cli.py  (2 of 2): the dispatch ----------
def run(args):
    lib = storage.load(args.db)
    log.debug("loaded %d tracks from %s", len(lib), args.db)
    if args.cmd == "add":
        lib.append(Track(args.title, args.artist, args.duration_s))
        storage.save_atomic(args.db, lib)                    # ch 16, every write
        print(f"added {args.title!r} -- the library now holds {len(lib)}")
    elif args.cmd == "list":
        for i, t in enumerate(lib, 1):
            print(f"{i:>2}. {t.title:<16}{t.artist:<15}{t.clock:>6}  {t.source}")
        print(f"-- {len(lib)} tracks, {sum(t.duration_s for t in lib)}s total")
    elif args.cmd == "search":
        hits = [t for t in lib if matches(t, args.needle)]
        for t in hits:
            print(f"{t.title} -- {t.artist} ({t.clock})")
        print(f"-- {len(hits)}/{len(lib)} matched {args.needle!r}")
    elif args.cmd == "export":
        n = storage.export_csv(args.out, lib)
        print(f"wrote {n} rows to {args.out}")
    elif args.cmd == "fetch":
        t = from_url(args.url)                               # no socket is opened
        lib.append(t)
        storage.save_atomic(args.db, lib)
        print(f"fetched {t.title!r} by {t.artist} ({t.clock}) -- source={t.source}")
    return 0


def main(argv=None):
    args = build_parser().parse_args(argv)
    logging.basicConfig(level=logging.DEBUG if args.verbose else logging.WARNING,
                        format="%(levelname)-8s %(name)s: %(message)s")
    log.debug("parsed %s", vars(args))
    return run(args)


# ---------- medialib/__main__.py ----------
"""__main__.py -- the front door. python -m medialib finds this by name."""
import sys
from medialib.cli import main

if __name__ == "__main__":          # a plain string comparison, nothing more
    sys.exit(main(sys.argv[1:]))


# ---------- the terminal: stand in medialib-project/ ----------
$ python -m medialib -h
$ python -m medialib add -h
$ python medialib/__main__.py list

$ python -m medialib -h
usage: medialib [-h] [--db DB] [-v] COMMAND ...

a tiny media library

positional arguments:
  COMMAND
    add          add one track by hand
    list         print the whole library
    search       find tracks by title or artist
    export       write the library out as CSV
    fetch        SIMULATED metadata fetch -- no network

options:
  -h, --help     show this help message and exit
  --db DB        where the library lives
  -v, --verbose  turn on DEBUG logging

$ python -m medialib add -h
usage: medialib add [-h] title artist duration_s

positional arguments:
  title
  artist
  duration_s

options:
  -h, --help  show this help message and exit

$ python medialib/__main__.py list
Traceback (most recent call last):
  File "C:\tmp\medialib-project\medialib\__main__.py", line 3, in <module>
    from medialib.cli import main
ModuleNotFoundError: No module named 'medialib'

When the line alone is not enough, stop scattering print statements and re-running. Drop breakpoint() right before the fault. It's the builtin that honors PYTHONBREAKPOINT and lands you in pdb. Now you are standing inside the frame with its locals still alive:

def next_song(playlist, current):
    n = len(playlist.songs)
    nxt = (current + 2) % n     # BUG: off by one — should be current + 1
    import pdb; pdb.set_trace()  # pdb> p current, n, nxt  — see the live state
    return playlist.songs[nxt]

Green and installable and startable. Then a user loads fifty thousand songs and the prompt freezes for four seconds on every save. Correct-but-unusable is still a failure they feel. Next: keeping the loop responsive, and fixing the right slow line. →

06Responsiveness and the hot path

The app is correct and proven, and then a user loads a fifty-thousand-song library and the prompt freezes solid for four seconds on every save, or the interface locks while a background scan runs. Correct-but-unusable is a failure the user feels in their fingers. Two distinct needs collide here, and the honest engineer keeps them separate. First, the command loop must stay answering while heavy work happens. Second, when one function is genuinely slow, you must fix the right one — because your intuition about which line is slow is, reliably, wrong. Keep the app responsive; and measure before you touch a single line for speed.

Responsiveness runs straight into the GIL, CPython's Global Interpreter Lock. CPython executes one bytecode stream at a time. The GIL is the single baton, and only the thread holding it runs Python. So a pure-Python CPU-bound loop in your command thread — rescoring fifty thousand songs — holds the baton the entire time, and no other thread can run Python until it finishes. Spawning threads will not make that loop faster and will not free the prompt during the compute. But here is the hinge: the GIL is released around blocking I/O syscalls. When a thread calls read, write, fsync, or a socket op, CPython drops the baton before sleeping in the kernel and reacquires it on return. While one thread waits on the disk, another thread runs Python. So the correct tool depends entirely on the kind of work:

import threading, multiprocessing

# I/O-bound (our fsync-heavy save): a thread KEEPS the loop responsive,
# because the worker sleeps in the syscall with the GIL released.
threading.Thread(target=save_atomic, args=(path, pl)).start()

# CPU-bound (rescoring 50k songs): a thread does NOT help — GIL held the
# whole time. Use a process: a second interpreter, its own GIL, real
# parallelism — paid for by pickling data across the process boundary.
with multiprocessing.Pool() as pool:
    scores = pool.map(score_song, library)   # args pickled out, results pickled back

There is the whole decision tree, and it is worth memorizing over any "just use threads" advice. If the heavy work is I/O — the fsync-heavy save, reading a big file — a background thread keeps the loop answering, because the worker is asleep in a syscall with the baton down. If the work is CPU-bound Python, threads are useless. You have two real options. One is multiprocessing: a second interpreter, its own GIL, genuine parallelism, paid for with pickling across the process boundary. That's the volume's law again: only bytes cross. The other is to make the loop cooperative, doing the work in bounded chunks and checking for input between them.

Now the second need: never optimize by eye. cProfile is a deterministic profiler. It registers a callback on CPython's C-level call/return trace hook, so every Python function call and return is timed. You read its output with intent, using two columns that mean genuinely different things. tottime is the time spent inside a function, excluding its callees. It finds the actual hot leaf. cumtime is the time including everything the function called. It finds the expensive subtree. Here is the classic, humbling surprise. The function you were sure was slow has a tiny tottime, and the real cost is a trivial three-line helper called two million times.

import cProfile, pstats

cProfile.run("save_and_rescore(library)", "prof.out")
st = pstats.Stats("prof.out")
st.sort_stats("tottime").print_stats(8)   # the hot LEAF — where time is actually burned
st.sort_stats("cumtime").print_stats(8)   # the expensive SUBTREE — who drags the most in
# the surprise: score_song has tiny tottime; a helper with ncalls=2,000,000 dominates.

Correct, installable, startable, provable, responsive, and profiled. That is a lot of claims. The last question is the hardest, because it has no error message: how do you know when to stop? →

07What 'done' means

Every hobby project shares one failure mode: it is 90% done forever. The code runs on the author's laptop, has a couple of commented-out tests, and no one else can start it. This section closes the hardest need in the book, precisely because it produces no error message. That need is knowing when to stop, and having an objective, checkable definition of done that a stranger could verify without asking you a single question. "Done" has to mean something a third party can confirm, or it means nothing at all. So we frame it not as a feeling but as a set of falsifiable claims. Each one is traceable to a boundary this volume taught you to cross, and each one is runnable.

# 1. INSTALLS from a lockfile — fresh venv reproduces the exact environment  (Ch 22)
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.lock        # matching versions + hashes, or it aborts

# 2. RUNS from one command — a real package with a __main__ entry point       (Ch 21)
python -m app --version                 # no cwd assumptions, no hidden setup

# 3. PROVES itself — model + engine + storage + the regression test        (Ch 24-25)
pytest -q                               # green, or you are not done

# 4. SURVIVES a power cut — kill mid-save, then the file still loads clean (Ch 16, 18)
python -m app save & sleep 0.05; kill -9 $!   # no cleanup handler runs
python -m app load                      # the LAST COMPLETE playlist, intact

# 5. STAYS usable under load — the one hot path was profiled and tuned     (Ch 26, 29)
python -m app profile-save              # tottime evidence, not a guess

Read those five claims slowly, because each is a promise you can break on purpose to check. Claim 1 — installs from a lockfile: a fresh venv plus pip install against the pinned lock reproduces the exact tree. You can verify it on a clean machine or fresh container, where the install succeeds with matching versions and hashes. Claim 2 — runs from one command: python -m app starts it with no working-directory assumptions, because section 4 made it a genuine package with a __main__. Claim 3 — proves itself: pytest is green and covers model, engine, storage, and the regression test from the debugging pass. So the claims that matter are mechanically checked rather than asserted in prose. Claim 4 — survives a power cut: because the save is atomic, you can kill -9 the process mid-fsync with no cleanup and confirm the file still loads as the last complete playlist. Claim 5 — stays usable under load: the loop answers during heavy work, and the one hot path was profiled and tuned with a before/after number, not a hunch.

# ---------- tests/test_medialib.py ----------
"""test_medialib.py -- one test per layer, each where that layer lives."""
from medialib import storage
from medialib.cli import main
from medialib.tracks import Track, matches, from_url


def test_track_is_a_value():                      # model: no disk, no clock
    a = Track("Titanium", "David Guetta", 245)
    assert a == Track("Titanium", "David Guetta", 245)
    assert a.clock == "4:05"


def test_search_is_case_folded():                 # model: still no I/O
    t = Track("Blinding Lights", "The Weeknd", 200)
    assert matches(t, "BLINDING") and not matches(t, "titanium")


def test_save_load_round_trips(tmp_path):         # storage: a throwaway directory
    db = str(tmp_path / "lib.json")
    lib = [Track("Titanium", "David Guetta", 245)]
    storage.save_atomic(db, lib)
    assert storage.load(db) == lib


def test_add_then_list(tmp_path, capsys):         # cli: argv in, stdout out
    db = str(tmp_path / "lib.json")
    assert main(["--db", db, "add", "Titanium", "David Guetta", "245"]) == 0
    main(["--db", db, "list"])
    assert "Titanium" in capsys.readouterr().out


def test_fetch_never_touches_the_network():       # the honest seam
    t = from_url("https://example.com/watch/blinding-lights")
    assert t.source == "web" and t.duration_s == 200


# ---------- pyproject.toml ----------
[project]
name = "medialib"
version = "1.0.0"
description = "a tiny media library, standard library only"
requires-python = ">=3.10"
dependencies = []

[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"


# ---------- the terminal: seven commands, in this order ----------
$ python -m medialib add "Blinding Lights" "The Weeknd" 200
$ python -m medialib -v add "Titanium" "David Guetta" 245
$ python -m medialib fetch https://example.com/watch/levels
$ python -m medialib list
$ python -m medialib search LIGHT
$ python -m medialib export -o starter.csv
$ python -m medialib add "Levels" "Avicii" three
$ python -m pytest -q

$ python -m medialib add "Blinding Lights" "The Weeknd" 200
added 'Blinding Lights' -- the library now holds 1

$ python -m medialib -v add "Titanium" "David Guetta" 245
DEBUG    medialib: parsed {'db': 'library.json', 'verbose': True, 'cmd': 'add', 'title': 'Titanium', 'artist': 'David Guetta', 'duration_s': 245}
DEBUG    medialib: loaded 1 tracks from library.json
added 'Titanium' -- the library now holds 2

$ python -m medialib fetch https://example.com/watch/levels
fetched 'Levels' by Avicii (3:23) -- source=web

$ python -m medialib list
 1. Blinding Lights The Weeknd       3:20  local
 2. Titanium        David Guetta     4:05  local
 3. Levels          Avicii           3:23  web
-- 3 tracks, 648s total

$ python -m medialib search LIGHT
Blinding Lights -- The Weeknd (3:20)
-- 1/3 matched 'LIGHT'

$ python -m medialib export -o starter.csv
wrote 3 rows to starter.csv

$ python -m medialib add "Levels" "Avicii" three
usage: medialib add [-h] title artist duration_s
medialib add: error: argument duration_s: invalid int value: 'three'
   (exit code 2)

$ python -m pytest -q
.....                                                                    [100%]
5 passed in 0.03s

# ---------- 1. medialib/cli.py -- three edits ----------

import argparse, logging, time                    # + time, for the ch 29 bracket

# ...in build_parser(), one line, above the fetch parser:
    sub.add_parser("stats", help="a one-screen summary of the library")

# ...in run(), one new branch, above the fetch branch:
    elif args.cmd == "stats":
        if not lib:
            print("the library is empty")         # the guard the empty case needs
            return 0
        t0 = time.perf_counter()                             # ch 29: the bracket
        total = sum(t.duration_s for t in lib)
        by = {}
        for t in lib:
            by[t.source] = by.get(t.source, 0) + 1
        longest = max(lib, key=lambda t: t.duration_s)
        print(f"{'tracks':<9}{len(lib):>7}")
        print(f"{'runtime':<9}{total // 60:>4}:{total % 60:02d}")   # ch 17: :02d
        for src, n in sorted(by.items()):
            print(f"{src:<9}{n:>7}{n / len(lib):>9.0%}")     # ch 17: the % type
        print(f"{'longest':<9}{longest.title} ({longest.clock})")
        log.debug("stats over %d tracks in %.3f ms", len(lib),
                  (time.perf_counter() - t0) * 1000)         # ch 25: lazy %


# ---------- 2. tests/test_medialib.py -- one more test ----------

def test_stats_counts_and_percentages(tmp_path, capsys):   # the new subcommand
    db = str(tmp_path / "lib.json")
    main(["--db", db, "add", "Titanium", "David Guetta", "245"])
    main(["--db", db, "fetch", "https://example.com/watch/levels"])
    capsys.readouterr()                                    # drop the add/fetch noise
    main(["--db", db, "stats"])
    out = capsys.readouterr().out
    assert "50%" in out and "7:28" in out and "Titanium" in out


# ---------- 3. the run, on the three-track library from the type-along ----------

$ python -m medialib stats
tracks         3
runtime    10:48
local          2      67%
web            1      33%
longest  Titanium (4:05)

$ python -m medialib -v stats
DEBUG    medialib: parsed {'db': 'library.json', 'verbose': True, 'cmd': 'stats'}
DEBUG    medialib: loaded 3 tracks from library.json
DEBUG    medialib: stats over 3 tracks in 0.025 ms
tracks         3
runtime    10:48
local          2      67%
web            1      33%
longest  Titanium (4:05)

$ python -m pytest -q
......                                                                   [100%]
6 passed in 0.05s


# ---------- the two skills, named ----------
#
#  ch 17, the format spec.  Every column is one spec, not one loop.
#      {'tracks':<9}   left-align in 9 columns      <  >  ^  are the whole alignment vocabulary
#      {total % 60:02d}  zero-pad to 2 digits       so 8 seconds prints as 08, never as 8
#      {n / len(lib):>9.0%}  the % TYPE             it multiplies by 100 and appends the sign
#          0.6666...  ->  "67%".  Written by hand that is round(n / len(lib) * 100) and a
#          str concat and a rjust -- three chances to be wrong, replaced by four characters.
#
#  ch 29, the bracket + ch 25, lazy logging.
#      t0 = perf_counter() ... perf_counter() - t0   is the ONLY honest stopwatch: monotonic,
#      ~100 ns, and only DIFFERENCES mean anything. time.time() would have reported 0.000000
#      here, because 25 microseconds is far under the Windows 15.6 ms tick.
#      log.debug("... %.3f ms", ...) does the formatting ONLY if DEBUG is enabled, so the
#      measurement is free in normal use and one flag away when you want it.
#
#
# ---------- question 1: the empty library ----------
#
#   max(lib, key=...) raises  ValueError: max() arg is an empty sequence.
#   sum() is fine (it returns 0) and the loop simply does not run -- max is the only
#   line that cannot survive an empty list. Hence the two-line guard at the top.
#   Note that `list` on an empty library already prints "-- 0 tracks, 0s total"
#   quite happily, which is exactly why the failure is easy to miss until a user finds it.
#
#
# ---------- question 2: where the arithmetic goes ----------
#
#   It went in cli.py, and that is defensible for a summary that exists only to be
#   printed: the numbers and their layout are one concern. It is also the FIRST thing
#   in this app that has grown a second reason to change -- the day you want the same
#   totals in `export`, or as JSON, the arithmetic has to move down into a new pure
#   module (say analytics.py, importing tracks and nothing else) with cli.py left
#   holding only the f-strings. storage stays the only module that touches the disk
#   either way: stats reads through storage.load and writes nothing at all.
#   The honest summary: correct today, and the exact place a sixth file will be born.

Here is the through-line to say out loud, the sentence this entire volume was built to earn: shipping is not one big skill — it is the sum of every boundary this volume taught you to cross. The module boundary: imports and sys.modules, from section 1. The type boundary: dataclasses generating code, and static checking reading the same hints, from section 2. The process/kernel boundary: the fsync and rename syscalls that turn a live object into durable bytes, from section 3. The environment boundary: venv and lockfile, from section 4. The correctness boundary: tests and pdb, from section 5. The performance boundary: the GIL and the profiler, from section 6. A program a stranger can trust is exactly a program where all six crossings are clean and each one is checkable.

# ---------- part 1: the twenty-minute list, ranked, with its check ----------
#
#  1. A README with ONE start command.                            (claim 2)
#       Three lines is enough: python -m medialib -h, one worked example, the
#       python version you tested on. CHECK: hand it to someone and watch them
#       start it without asking you a question. That is the entire test.
#
#  2. --version, because the done-list already promises it.       (claim 2)
#       p.add_argument("--version", action="version",
#                      version=f"%(prog)s {medialib.__version__}")
#       CHECK: python -m medialib --version   ->  medialib 1.0.0, exit 0.
#
#  3. Pin the environment and write down the dev dependency.      (claim 1)
#       requires-python = ">=3.10" is a floor, not a pin. Record the interpreter
#       you actually tested (3.12.7) and put pytest where a stranger can find it:
#         [project.optional-dependencies]
#         dev = ["pytest==7.4.4"]
#       CHECK: python -m venv .venv ; python -m pip install -e ".[dev]" ; pytest -q
#       in a directory that is NOT yours. The runtime lock is nearly empty because
#       we used no third-party runtime deps at all -- that is a durability claim,
#       not an accident, and it belongs in the README.
#
#  --- past twenty minutes: written down as debt, not shipped tonight ---
#
#  4. Turn the four crashes in part 2 into usage errors.          (claim 3)
#  5. A directory fsync after os.replace, guarded for Windows.    (claim 4)
#  6. A kill -9 mid-save test, and a hot-path measurement.        (claims 4, 5)
#
#
# ---------- part 2: run it hostilely. four claims, four failures. ----------
#
#  CLAIM 2 -- "runs from one command"          FALSE
#
#    $ python -m medialib --version
#    usage: medialib [-h] [--db DB] [-v] COMMAND ...
#    medialib: error: the following arguments are required: COMMAND
#    (exit 2)
#
#    Read that twice. It did not say "unknown option --version". required=True on
#    add_subparsers fires FIRST, so the message names the wrong problem entirely.
#    A usage line, correctly -- but a misleading one. This is the failure that was
#    sitting in plain sight: done.sh runs this exact command in claim 2.
#
#  CLAIM 3 -- "proves itself"                  INCOMPLETE
#
#    $ python -m medialib fetch https://example.com/watch/nope
#      File ".../medialib/tracks.py", line 32, in from_url
#        title, artist, secs = CATALOGUE[slug]
#    KeyError: 'nope'
#
#    A raw traceback with our file paths in it, for a typo the user made. They
#    deserve:  medialib fetch: error: no metadata for 'nope'  and exit 2.
#    The suite is green because not one of the five tests calls from_url with a
#    slug that is missing. Green means five claims held. It never meant correct.
#
#    Same disease, second door:
#    $ python -m medialib --db nosuchdir/lib.json add A B 1
#    FileNotFoundError: [Errno 2] No such file or directory: '...\tmpjfzm4qb0.tmp'
#    mkstemp cannot create a temp file in a folder that does not exist. The user
#    typed a path; they should get a sentence about that path, not an Errno.
#
#  CLAIM 4 -- "survives a power cut"           PARTLY, AND UNPROVEN
#
#    save_atomic does write-tmp -> flush -> fsync -> os.replace. What it never does
#    is fsync the containing DIRECTORY, so on POSIX the rename itself can still be
#    lost. Section 3 said both halves are required. And nothing tests it: every
#    test runs on a machine that does not lose power at the cruel microsecond.
#    CHECK: start a save in a subprocess, kill -9 mid-write, reload in the parent,
#    assert you got the last COMPLETE library. Until that test exists, claim 4 is
#    a belief.
#
#  CLAIM 5 -- "stays usable under load"        UNMEASURED
#
#    Every write rewrites the whole JSON file and fsyncs it. At three tracks that
#    is invisible; at fifty thousand it is a full re-serialise per `add`. We have
#    not measured it once, so we cannot claim it. python -m cProfile -s tottime
#    -m medialib add ... on a large library is the twenty-minute answer, and the
#    number it prints decides whether there is anything to fix at all.
#
#
# ---------- part 3: the ranking, and why ----------
#
#   SHIP:  README, --version, the pinned dev dependency.
#     All three serve the same person: a stranger who has to install, start, and
#     verify without you in the room. They cost minutes and they are the only
#     items that change whether the app can be handed over AT ALL.
#
#   DEBT:  the three crash-to-usage fixes, the directory fsync, the kill -9 test,
#          the profile of a large library.
#     Each is real, each is cheap alone, and none of them blocks the handover.
#     Write them down where the next person can see them -- because the difference
#     between debt and a bug you forgot is whether it is written down.
#
#   And the one thing NOT on either list: new features. `stats` was fun. It is
#   also the exact move that keeps a project 90% done forever. Done is not a
#   feeling; it is five claims that are currently, verifiably true.

That is Volume 2 — the physics of every boundary a process must cross to be useful. You began inside one process where every name pointed straight at a real thing; you end able to flatten that thing to bytes and rebuild it faithfully anywhere. The exercises are over. You have shipped. →