24Testing — what green actually proves

In Chapter 23 we watched the type checker judge our code without ever running it — static hints, settled before line 1 executes. In this one we actually run the thing and ask a harder question: does it do the right thing? Picture the moment. You changed one function, and your stomach dropped. Did you just break the three callers you forgot existed? A test is the machine that answers that, without you re-checking by hand. Here's the plan. We take pytest apart to the metal: how it finds your claims by walking the filesystem, how it rewrites assert at the syntax-tree level so a failure explains itself, how fixtures inject dependencies, and how parametrize multiplies one body into many. And the whole way through we keep asking the one thing that matters — when the bar goes green, what has that green really proven? By the end you can pin a claim like total([200,245]) is 445 to your code and have a machine re-check it forever. You'll know exactly, and honestly, what that green does and doesn't buy you.

01A test is an executable claim

Let's start with the fear, because the fear is the whole reason testing exists. You changed total() to handle a new edge case, and now you're afraid. Somewhere in the codebase three functions call it, and you can't remember which. So you do what everyone does the first year. You run the program by hand, type a few inputs, squint at the output, and ship on the fact that nothing looked wrong. That works for a weekend. It does not survive a month. The number of things you'd have to re-check by hand grows faster than any afternoon, so in practice you check almost nothing and call it confidence. The felt need is brutally simple: write down, once, a sentence like total([200,245]) is 445, and have a machine re-check that sentence forever. A test is not a chore bolted onto real work. It is the only way to keep a claim about your code alive after the moment you first believed it.

Now throw the word 'test' away and look at the primitive underneath it: assert. Watch what it actually is — not a function, but a statement the compiler itself knows about, and it compiles down to something you could have written by hand. assert x == 5 becomes, in bytecode, almost exactly if not (x == 5): raise AssertionError. Watch the machine do it:

import dis
dis.dis(compile("assert x == 5", "<s>", "exec"))

# LOAD_NAME            x
# LOAD_CONST           5
# COMPARE_OP           ==
# POP_JUMP_IF_TRUE     --> end     # true? jump clean over the raise
# LOAD_ASSERTION_ERROR             # false? build the exception
# RAISE_VARARGS        1           # and throw it
# end: LOAD_CONST None / RETURN_VALUE

Read those six lines and the whole mystery of testing evaporates. An assert evaluates an expression. If it is truthy, control jumps clean over the raise and the function keeps going, but if it is falsy, it constructs an AssertionError and throws. Two exact truths ride along. First, assert takes a second operand: assert expr, msg. The message becomes the exception's argument, so a failure can carry a sentence. Second, and this is the one that decides where you're allowed to use it: asserts are compiled entirely out when Python runs with -O. The name __debug__ is a compile-time constant, true normally and false under -O, so under optimization assert emits zero bytecode. That is why production code must never validate untrusted input with assert. Flip on -O and your check simply isn't there. But a test is debug-time-only code by definition. Assert is not merely allowed there, it is exactly right.

So here is the definition the rest of the chapter builds on. A test is just a function whose body makes claims with assert. It takes no arguments you supply by hand. It returns None on success, and raises on failure. 'The test passed' means, with no magic whatsoever, the function returned without raising. pytest does not inspect your soul. It calls your function inside a try/except. If the call returns, it paints the dot green. If any exception propagates out — an AssertionError you wrote, or a TypeError you didn't expect — it catches it and paints the dot red.

def total(items):
    return sum(items)

def test_total():
    assert total([200, 245]) == 445

# pytest adds nothing but the calling loop — you can BE pytest:
try:
    test_total()
    print("green")          # returned → passed
except AssertionError as e:
    print("red", repr(e))  # raised → failed

That try/except around a function call is the entire runner in miniature. Everything downstream is a refinement of it. Collection is the question 'which functions do I call?'. The assert rewrite is 'how do I make the raised exception informative?'. Fixtures are 'what do I pass in as arguments?'. Hold the exception-as-signal model in your head and none of pytest will ever feel like sorcery again.

You can write a claim. But you never told pytest your ten functions exist — no registry, no list, no main(). So how does it find them? Next: collection, the search pytest runs against your filesystem. →

02Collection: discovery by convention

You wrote ten test functions across four files. You never wired them together. There is no list of tests, no registry, no main() that calls them in order. Yet you type pytest and it finds all ten and runs them. A program you never told about your functions somehow knows they exist. And when a colleague insists the file be named test_orders.py and the function test_ships, it feels like superstition — some etiquette the tool enforces for no reason. The need here is to see that the naming rules are not etiquette. They are the literal query pytest runs against your disk and your module objects. Once you see collection as a search with rules, the worst failure in testing is a test that silently never runs. It stops being a mystery you shrug at and becomes something you can predict exactly.

Collection is a two-stage reflection process, and both stages are pure convention matched against real runtime objects. Stage one is the filesystem. pytest walks the directory tree from the rootdir (respecting testpaths if you set it) and selects files whose names match test_*.py or *_test.py. A file called helpers.py is invisible to this walk no matter how many asserts it contains. Stage two is import plus introspection. For each matched file, pytest imports it as a module — and this is the load-bearing detail of the entire section: importing a module runs it top to bottom. Every line at module level executes during collection, before a single test runs. That is why an import error, or a top-level line that raises, surfaces as a collection error — pytest never got far enough to call any test, because it crashed while reading the file.

import re, importlib

# stage one: filesystem — keep only files matching the convention
candidates = ["test_orders.py", "helpers.py", "cart_test.py"]
matched = [f for f in candidates
           if re.match(r"test_.*\.py$|.*_test\.py$", f)]
# ['test_orders.py', 'cart_test.py']  — helpers.py is simply not seen

# stage two: import each (RUNS it), then walk its namespace
module = importlib.import_module("test_orders")
items = [obj for name, obj in vars(module).items()
         if name.startswith("test") and callable(obj)]

After the import, pytest walks the module's namespace with the same machinery as dir() and vars() — it introspects the live module object. It plucks out functions whose name starts with test, and classes named Test* (with no __init__) whose methods start with test. Each match becomes a Test item: an object pytest will later call. This is why a mis-named function vanishes without a word. def check_total() is a perfectly good function that asserts perfectly good things — and pytest walks right past it, because check does not start with test. No error, no warning. It simply never runs, and your green bar is green over a claim that was never checked.

you type
# ---------- myproject/tracks.py ----------
"""Duration helpers for the jukebox."""

LIBRARY = [("Blinding Lights", 200), ("Titanium", 245), ("Levels", 340)]


def as_clock(seconds):
    minutes, secs = divmod(seconds, 60)
    return f"{minutes}:{secs:02d}"


def total(tracks):
    return sum(secs for _title, secs in tracks)


# ---------- myproject/tests/test_tracks.py ----------
from tracks import LIBRARY, as_clock, total


def test_total_sums_the_library():
    shelf   = LIBRARY               # ARRANGE
    seconds = total(shelf)          # ACT
    assert seconds == 785           # ASSERT


def test_as_clock_pads_the_seconds():
    assert as_clock(245) == "4:05"


def check_total():                  # not 'test' -- pytest never sees it
    assert total([]) == 999


# ---------- myproject/tests/helpers.py ----------
assert 1 == 2      # never runs: helpers.py does not match test_*.py


# ---------- the terminal, in the Ch 22 venv, standing in myproject/ ----------
$ python -m pip install pytest
$ python -m pytest --version
$ python -m pytest
$ python -m pytest --collect-only -q
$ python -m pytest -q -k "clock"
$ echo $?
$ pytest                              # the console script. NOT the same command.

you see
$ python -m pip install pytest
Collecting pytest
  Downloading pytest-9.1.1-py3-none-any.whl.metadata (7.6 kB)
Collecting colorama>=0.4 (from pytest)
Collecting iniconfig>=1.0.1 (from pytest)
Collecting packaging>=22 (from pytest)
Collecting pluggy<2,>=1.5 (from pytest)
Collecting pygments>=2.7.2 (from pytest)
Installing collected packages: pygments, pluggy, packaging, iniconfig,
    colorama, pytest
Successfully installed colorama-0.4.6 iniconfig-2.3.0 packaging-26.3
    pluggy-1.6.0 pygments-2.20.0 pytest-9.1.1

$ python -m pytest --version
pytest 9.1.1

$ python -m pytest
============================= test session starts =============================
platform win32 -- Python 3.12.7, pytest-9.1.1, pluggy-1.6.0
rootdir: C:\tmp\ch24lang\d_anat
collected 2 items

tests\test_tracks.py ..                                                  [100%]

============================== 2 passed in 0.00s ==============================

$ python -m pytest --collect-only -q
tests/test_tracks.py::test_total_sums_the_library
tests/test_tracks.py::test_as_clock_pads_the_seconds

2 tests collected in 0.00s

$ python -m pytest -q -k "clock"
.                                                                        [100%]
1 passed, 1 deselected in 0.00s

$ echo $?
0

$ pytest
=================================== ERRORS ====================================
____________________ ERROR collecting tests/test_tracks.py ____________________
ImportError while importing test module '...\d_anat\tests\test_tracks.py'.
Traceback:
tests\test_tracks.py:1: in <module>
    from tracks import LIBRARY, as_clock, total
E   ModuleNotFoundError: No module named 'tracks'

There is one subtlety worth meeting precisely, because it bites everyone once. In pytest's default 'prepend' import mode, it inserts each test file's directory onto sys.path and imports the file by its bare basename. So two files both named test_utils.py in different folders collide. Python's import system sees the same module name twice, and one shadows the other, producing a baffling error. The fixes are exactly what the mechanism predicts. Add __init__.py files so the packages give the modules distinct dotted names. Or switch to importmode=importlib, which imports by full path and never touches sys.path.

Now the payoff that ties collection to section 1. Each collected item is run as its own function call inside its own try/except. When item three raises an AssertionError, that exception unwinds item three's stack and is caught by the runner. The loop simply moves to item four. One test's failure cannot poison the next, because they were never on the same stack. Contrast a hand-rolled main() that calls your tests in sequence. There, the first uncaught exception kills every test after it, and you learn about one failure while five real ones hide behind it. Isolation is not a mystical property of tests. It is a structural gift pytest hands you by choosing to run N independent calls instead of one long function.

pytest can find your claim and run it in isolation. But when assert response.status == 200 fails, plain Python tells you only 'AssertionError' — not the value it saw. Next: the syntax-tree surgery that makes failures talk. →

03The assert rewrite makes failures talk

You wrote assert response.status == expected. It fails. Plain Python hands you exactly one bit of information: AssertionError. Not the value of status, not the value of expected, nothing you can act on. This single poverty is why an entire vocabulary exists. There's assertEqual(a, b), assertGreater, assertIn, assertAlmostEqual — a zoo of methods you memorize purely so the failure message will finally show you the two numbers. The need is to get the informative failure without the ceremony. And pytest's answer is genuinely startling the first time you see it. A bare assert a == b fails with assert 404 == 200 — both operands, printed. That is not normal Python. It cannot be, because you saw normal Python print nothing but the exception's name. Understanding the mechanism behind it dissolves the entire assertEqual family at once, and explains why the magic works only inside pytest.

The trick is compile-time surgery on the syntax tree, and it happens at import, not at call. Here is the exact pipeline. Before Python source becomes bytecode, it is parsed into an Abstract Syntax Tree. That's a tree of nodes where assert a == b is an Assert node wrapping a Compare node. pytest registers an import hook, a finder on sys.meta_path. So when a module pytest wants to collect is imported, pytest intercepts the import, reads the source, parses it to an AST, and runs a transformer over it. That transformer rewrites every Assert node into something much larger.

# you wrote:
assert get_status() == 200

# pytest rewrites it to roughly this, at import time:
_left  = get_status()          # evaluate each operand ONCE, into a temp
_right = 200
_ok    = _left == _right
if not _ok:
    raise AssertionError(
        f"assert {_left!r} == {_right!r}")   # reprs of the saved values

Read what that buys. The operands are saved into hidden temporaries, so each subexpression is evaluated exactly once. Even get_status(), which might have a side effect, is not called twice. And because their runtime values are sitting in _left and _right, the explanation can repr() them into a message that shows you precisely what pytest saw. The transformed AST is compiled to a .pyc cached under __pycache__ with a pytest-specific tag. So this rewrite cost is paid once per file, not once per run.

you type
# ---------- test_talk.py ----------
STOCK = {"Blinding Lights": 200, "Titanium": 245, "Levels": 340}


def status():
    return 404


def test_int():
    assert status() == 200


def test_str():
    assert "Titanuim" == "Titanium"


def test_list():
    assert [200, 245, 300] == [200, 245, 301]


def test_dict():
    assert STOCK == {"Blinding Lights": 200, "Titanium": 245, "Levels": 341}


def test_in():
    assert "Wake Me Up" in STOCK


# ---------- plain.py -- the same claim, outside pytest ----------
def status():
    return 404

try:
    assert status() == 200
except AssertionError as e:
    print("plain python  :", repr(e))


# ---------- the terminal ----------
$ python plain.py
$ python -m pytest -q

you see
$ python plain.py
plain python  : AssertionError()

$ python -m pytest -q
FFFFF                                                                    [100%]
================================== FAILURES ===================================
__________________________________ test_int ___________________________________

    def test_int():
>       assert status() == 200
E       assert 404 == 200
E        +  where 404 = status()

test_talk.py:9: AssertionError
__________________________________ test_str ___________________________________

    def test_str():
>       assert "Titanuim" == "Titanium"
E       AssertionError: assert 'Titanuim' == 'Titanium'
E
E         - Titanium
E         ?       -
E         + Titanuim
E         ?      +

test_talk.py:13: AssertionError
__________________________________ test_list __________________________________

    def test_list():
>       assert [200, 245, 300] == [200, 245, 301]
E       assert [200, 245, 300] == [200, 245, 301]
E
E         At index 2 diff: 300 != 301
E         Use -v to get more diff

test_talk.py:17: AssertionError
__________________________________ test_dict __________________________________

    def test_dict():
>       assert STOCK == {"Blinding Lights": 200, "Titanium": 245, "Levels": 341}
E       AssertionError: assert {'Blinding Li...'Levels': 340} == {'Blinding Li...'Levels': 341}
E
E         Omitting 2 identical items, use -vv to show
E         Differing items:
E         {'Levels': 340} != {'Levels': 341}
E         Use -v to get more diff

test_talk.py:21: AssertionError
___________________________________ test_in ___________________________________

    def test_in():
>       assert "Wake Me Up" in STOCK
E       AssertionError: assert 'Wake Me Up' in {'Blinding Lights': 200, 'Titanium': 245, 'Levels': 340}

test_talk.py:25: AssertionError
5 failed in 0.03s

Three exact consequences fall out of 'it's an import-time transform', and each one explains a real confusion. One: it only fires for files pytest imports through its hook. An assert buried in a plain helper module you import normally is not rewritten. It gives you the bare, useless AssertionError, unless you explicitly call pytest.register_assert_rewrite('mymodule') before it's imported. Two: a module already imported before pytest's hook installs won't be rewritten at all, so ordering matters. Three: the introspection is type-aware. For == on two lists, dicts, or multiline strings, pytest emits a structured diff — the first differing element, or a unified diff — driven by the operands' runtime types:

def test_playlist():
    assert [200, 245, 300] == [200, 245, 301]

# pytest's failure isn't just 'AssertionError' — it points at the culprit:
#   assert [200, 245, 300] == [200, 245, 301]
#     At index 2 diff: 300 != 301

Now land the payoff. assertEqual, assertGreater, and the rest of that zoo exist because vanilla unittest could not see inside the expression. By the time self.assertTrue(a == b) runs, the comparison has already collapsed to a lone True or False. The operands are gone. So unittest made you name the relation, assertEqual, precisely so it could receive a and b separately and print them. pytest never has that problem. It sees the whole expression tree at import time, so one operator — ==, <, in, is — plus the rewrite reconstructs a better message than any hand-written assertEqual ever managed. You get to write the plainest possible Python and receive the richest possible failure. That is not a convenience layer bolted on top. It is a smarter place to stand.

you type
# ---------- tracks.py (the ch 13 guards, still unchanged) ----------
class PlaylistError(Exception):
    """Something is wrong with this playlist."""


def add_song(shelf, title, seconds):
    if seconds <= 0:
        raise ValueError(f"impossible length: {seconds}s")
    if any(t == title for t, _s in shelf):
        raise PlaylistError(f"{title!r} is already on the shelf")
    shelf.append((title, seconds))
    return shelf


# ---------- test_errors.py ----------
import pytest

from tracks import PlaylistError, add_song


def test_duplicate_raises():
    shelf = [("Titanium", 245)]
    with pytest.raises(PlaylistError):
        add_song(shelf, "Titanium", 245)


def test_message_matches():
    with pytest.raises(PlaylistError, match=r"already on the shelf"):
        add_song([("Titanium", 245)], "Titanium", 245)


def test_capture_the_object():
    with pytest.raises(ValueError) as excinfo:
        add_song([], "Titanium", -50)
    assert str(excinfo.value) == "impossible length: -50s"
    assert excinfo.type is ValueError


def test_the_shelf_is_untouched():
    shelf = [("Titanium", 245)]
    with pytest.raises(PlaylistError):
        add_song(shelf, "Titanium", 245)
    assert shelf == [("Titanium", 245)]


def test_forgot_to_break_it():
    with pytest.raises(PlaylistError):
        add_song([], "Levels", 340)          # this call SUCCEEDS


def test_wrong_class():
    with pytest.raises(PlaylistError):
        add_song([], "Levels", -1)           # raises ValueError, not PlaylistError


# ---------- the terminal ----------
$ python -m pytest -q

you see
$ python -m pytest -q
....FF                                                                   [100%]
================================== FAILURES ===================================
___________________________ test_forgot_to_break_it ___________________________

    def test_forgot_to_break_it():
>       with pytest.raises(PlaylistError):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       Failed: DID NOT RAISE PlaylistError

test_errors.py:32: Failed
______________________________ test_wrong_class _______________________________

    def test_wrong_class():
        with pytest.raises(PlaylistError):
>           add_song([], "Levels", -1)           # raises ValueError, not PlaylistError
            ^^^^^^^^^^^^^^^^^^^^^^^^^^

test_errors.py:38:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

shelf = [], title = 'Levels', seconds = -1

    def add_song(shelf, title, seconds):
        if seconds <= 0:
>           raise ValueError(f"impossible length: {seconds}s")
E           ValueError: impossible length: -1s

tracks.py:25: ValueError
=========================== short test summary info ===========================
FAILED test_errors.py::test_forgot_to_break_it - Failed: DID NOT RAISE Playli...
FAILED test_errors.py::test_wrong_class - ValueError: impossible length: -1s
2 failed, 4 passed in 0.03s

Failures talk now. But every test still opens the same database, builds the same temp dir, logs in the same client — ten lines of setup drowning the one line of claim. Next: fixtures, and dependency injection by name. →

04Fixtures are dependency injection

Every test in the file needs the same thing before it can start: a fresh database connection, a temp directory, a logged-in client. So the top of every test function grows the same six lines of setup and the same four lines of teardown, and now the boilerplate outweighs the claim. Worse, when the setup changes you edit it in twenty places. And if you get clever and build the object once to share it, test three mutates it and test seven fails. But only when they run in that order, which is the most maddening bug there is. The need is precise. Declare 'this test needs a fresh X', and have the machine build a clean X, hand it over, and tear it down afterward — with no boilerplate in the test body, and no test bleeding state into the next. That mechanism is dependency injection, and pytest's version of it is the feature people most misread as magic.

A fixture is a function decorated with @pytest.fixture. The wiring is pure name-matching through introspection. Before pytest calls a test, it inspects the test function's parameter names with inspect.signature. For each parameter, it looks up a fixture of that exact name, calls it, and passes the return value as that argument. That is the whole of 'injection'. The test declares a dependency by naming it as a parameter, and pytest resolves the name to a provider. Nothing is global, and the test never constructs the object itself.

import pytest

@pytest.fixture
def playlist():
    return [200, 245]        # Blinding Lights, Titanium

def test_total(playlist):        # names 'playlist' → pytest injects it
    assert total(playlist) == 445

Fixtures can themselves take fixture parameters, so pytest builds a dependency graph and resolves it depth-first. A db fixture that names a config parameter forces config to be built first, its result fed into db, and db's result fed into the test. Now the two mechanisms that matter most. Setup and teardown ride on yield. A fixture that yields is a generator. pytest calls next() to run everything up to the yield — that's setup — and hands the yielded value to the test. When the test finishes, pytest calls next() again, driving the generator past the yield to StopIteration. That runs everything after the yield, which is teardown. The yield is the literal seam between the two phases. And because pytest drives that second next() inside a finally, teardown runs even if the test failed.

@pytest.fixture
def db():
    conn = connect()        # setup — runs before the test
    yield conn              # <- the seam: hand conn to the test
    conn.close()            # teardown — runs after, even on failure

Scope controls how long the built object lives. The default, scope='function', means the fixture is re-called and re-torn-down for every single test — a fresh object each time. This is exactly why order-independence is the default. Test seven can't be poisoned by test three mutating the object, because test seven received a different object, built by a fresh call. Widen to scope='module' or 'session' and pytest calls the fixture once and memoizes the return value, reusing the same object across many tests. That is faster, since you connect to the database once, not four hundred times. But now shared mutable state is possible again, and you must understand the trade you are buying. Speed on one side, the risk of order-dependent tests on the other.

you type
# ---------- conftest.py ----------
import pytest


@pytest.fixture
def playlist():
    print("   [setup]  building a fresh playlist")
    return [200, 245]


# ---------- test_fix.py ----------
import pytest


@pytest.fixture
def logfile(tmp_path):
    path = tmp_path / "plays.log"
    path.write_text("Blinding Lights\n", encoding="utf-8")
    yield path                              # <- the seam: the test runs here
    print(f"   [teardown] {path.name} had {len(path.read_text())} bytes")


def test_a_mutates_its_own(playlist):
    playlist.append(999)
    assert playlist == [200, 245, 999]


def test_b_sees_a_fresh_one(playlist):
    assert playlist == [200, 245]           # the 999 never happened here


def test_the_logfile_is_real(logfile, tmp_path):
    logfile.write_text("Titanium\n", encoding="utf-8")
    print("   [test]   tmp_path =", tmp_path.name)
    assert logfile.read_text(encoding="utf-8") == "Titanium\n"
    assert logfile.parent == tmp_path


def test_gets_its_own_tmp_path(tmp_path):
    print("   [test]   tmp_path =", tmp_path.name)
    assert not (tmp_path / "plays.log").exists()


# ---------- the terminal ----------
$ python -m pytest -q -s
$ python -m pytest --fixtures-per-test test_fix.py::test_the_logfile_is_real

you see
$ python -m pytest -q -s
   [setup]  building a fresh playlist
.   [setup]  building a fresh playlist
.   [test]   tmp_path = test_the_logfile_is_real0
.   [teardown] plays.log had 9 bytes
   [test]   tmp_path = test_gets_its_own_tmp_path0
.
4 passed in 0.03s

$ python -m pytest --fixtures-per-test test_fix.py::test_the_logfile_is_real
------------------ fixtures used by test_the_logfile_is_real ------------------
------------------------------ (test_fix.py:22) -------------------------------
logfile -- test_fix.py:5
    no docstring available
tmp_path -- ...\_pytest\tmpdir.py:291
    Return a temporary directory (as :class:`pathlib.Path` object)
    which is unique to each test function invocation.
tmp_path_factory -- ...\_pytest\tmpdir.py:276
    Return a :class:`pytest.TempPathFactory` instance for the session.

Two final threads. First, connect fixtures to the chapter's spine: collection decides which functions to call, and fixtures decide what to pass to them. The parameter name is the whole interface between the two. Second, where do fixtures live so many files can share them? They live in conftest.py. pytest auto-discovers fixtures defined there for every test in that directory tree, with no import needed and no wiring. It is the exact same convention-over-configuration idea that powered collection. You never register a fixture any more than you register a test — you name it, put it in the right place, and the search finds it.

# ---------- jukebox_store.py -- chapter 18's store, with the run script stripped off ----------
import json
from dataclasses import dataclass, asdict
from pathlib import Path

DB = Path("jukebox.json")
VERSION = 1


@dataclass
class Track:
    title: str
    artist: str
    seconds: int
    plays: int = 0


def save(tracks, path=DB):
    """Whole library, one JSON document. Returns how many tracks landed."""
    payload = {"version": VERSION, "tracks": [asdict(t) for t in tracks]}
    with open(path, "w", encoding="utf-8", newline="\n") as f:
        json.dump(payload, f, indent=2, ensure_ascii=False)
    return len(tracks)


def load(path=DB):
    """JSON back into Track objects. A broken file is reported, never guessed at."""
    if not path.exists():
        return []
    with open(path, "r", encoding="utf-8") as f:
        try:
            payload = json.load(f)
        except json.JSONDecodeError as e:
            print(f"  {path.name} is not valid JSON: {e.msg} at char {e.pos}")
            return []
    if payload.get("version") != VERSION:
        print(f"  version {payload.get('version')} file, this code speaks {VERSION}")
    return [Track(**row) for row in payload["tracks"]]


# ---------- test_the_jukebox_store.py ----------
import json

import pytest

from jukebox_store import Track, load, save


@pytest.fixture
def library():
    """The three tracks, rebuilt fresh for every single test."""
    return [
        Track("Blinding Lights", "The Weeknd", 200, plays=1),
        Track('Sing, Sing, Sing | "Live"', "Benny Goodman", 512),
        Track("Bjork - Joga", "Bjork", 302),
    ]


@pytest.fixture
def db(tmp_path):
    """A path inside pytest's own temp dir. Nothing exists there yet."""
    return tmp_path / "jukebox.json"


def test_save_writes_the_file_and_counts(library, db):
    assert not db.exists()
    assert save(library, db) == 3
    assert db.exists()


def test_round_trip_returns_equal_tracks(library, db):
    save(library, db)
    assert load(db) == library


def test_the_delimiter_row_survives(library, db):
    save(library, db)
    assert load(db)[1].title == 'Sing, Sing, Sing | "Live"'


def test_load_of_a_missing_file_is_empty(db):
    assert load(db) == []


def test_the_file_is_readable_json_on_disk(library, db):
    save(library, db)
    payload = json.loads(db.read_text(encoding="utf-8"))
    assert payload["version"] == 1
    assert payload["tracks"][0]["plays"] == 1


def test_each_test_gets_its_own_directory(db):
    save([Track("Levels", "Avicii", 340)], db)
    assert len(load(db)) == 1

$ python -m pytest test_the_jukebox_store.py -v
============================= test session starts =============================
platform win32 -- Python 3.12.7, pytest-9.1.1, pluggy-1.6.0
rootdir: C:\tmp\ch24lang\lab
collecting ... collected 6 items

test_the_jukebox_store.py::test_save_writes_the_file_and_counts PASSED   [ 16%]
test_the_jukebox_store.py::test_round_trip_returns_equal_tracks PASSED   [ 33%]
test_the_jukebox_store.py::test_the_delimiter_row_survives PASSED        [ 50%]
test_the_jukebox_store.py::test_load_of_a_missing_file_is_empty PASSED   [ 66%]
test_the_jukebox_store.py::test_the_file_is_readable_json_on_disk PASSED [ 83%]
test_the_jukebox_store.py::test_each_test_gets_its_own_directory PASSED  [100%]

============================== 6 passed in 0.07s ==============================

$ python -m pytest -q -s -k "own_directory"     # where does tmp_path actually point?
tmp_path = C:\Users\ajair\AppData\Local\Temp\pytest-of-ajair\pytest-2\test_show0
.
1 passed in 0.01s

One fixture feeds one object to one test. But you want to run the same claim over empty lists, single elements, negatives, and huge inputs — without copy-pasting the body four times. Next: parametrize, one body into many cases. →

05Parametrize: one body, many cases

You want to test total() on the empty list, one element, some negatives, and a big list. So you copy the test function four times, changing two literals in each. Now the assertion logic lives in four places. Fix a bug in how you assert and you fix it four times. And when case three fails, the traceback says test_total_negatives, and you have to go read the body to remember what input that even was. The obvious escape is a loop inside one test, for case in cases: assert .... But that's worse, because the first failing case raises and unwinds the whole function. So you learn about failure one and nothing about two through four, and the whole thing reports as a single red. The need: run one claim over a table of inputs, where each row is its own independently-reported case that runs even if its neighbors fail — without duplicating the body. That is parametrization, and it is the difference between a test and a test suite.

import pytest

@pytest.mark.parametrize("items,expected", [
    ([],            0),
    ([200],         200),
    ([200, 245],    445),
    ([-5, 5],       0),
])
def test_total(items, expected):
    assert total(items) == expected

Here is the mechanism, and it is not what it looks like. @pytest.mark.parametrize is not a runtime loop. It acts at collection time. The mark attaches metadata to the test function. During collection — the same stage from section 2 that turned files into items — pytest reads that metadata and multiplies the single function into N distinct test items, one per row of the table, binding the named parameters to that row's values. It uses the very same parameter-injection machinery as fixtures. Parametrize is essentially a fixture whose values you supply explicitly. So the four-row table above does not produce one test that loops four times. It produces four separate items.

you type
# ---------- test_param.py ----------
import pytest

from tracks import as_clock, total


@pytest.mark.parametrize("items,expected", [
    ([],                          0),
    ([("Titanium", 245)],       245),
    ([("a", 200), ("b", 245)],  445),
    ([("a", -5), ("b", 5)],       0),
])
def test_total(items, expected):
    assert total(items) == expected


@pytest.mark.parametrize("seconds,expected", [
    pytest.param(0,    "0:00",  id="zero"),
    pytest.param(59,   "0:59",  id="under-a-minute"),
    pytest.param(60,   "1:00",  id="exactly-a-minute"),
    pytest.param(3600, "60:00", id="an-hour"),
])
def test_as_clock(seconds, expected):
    assert as_clock(seconds) == expected


@pytest.mark.parametrize("secs",   [0, 59, 245])
@pytest.mark.parametrize("prefix", ["", "~ "])
def test_stacked(prefix, secs):
    assert (prefix + as_clock(secs)).endswith(as_clock(secs))


# ---------- the terminal ----------
$ python -m pytest --collect-only -q
$ python -m pytest -q
$ python -m pytest -q -k "an-hour"
$ python -m pytest -q "test_param.py::test_total[items2-445]"

you see
$ python -m pytest --collect-only -q
test_param.py::test_total[items0-0]
test_param.py::test_total[items1-245]
test_param.py::test_total[items2-445]
test_param.py::test_total[items3-0]
test_param.py::test_as_clock[zero]
test_param.py::test_as_clock[under-a-minute]
test_param.py::test_as_clock[exactly-a-minute]
test_param.py::test_as_clock[an-hour]
test_param.py::test_stacked[-0]
test_param.py::test_stacked[-59]
test_param.py::test_stacked[-245]
test_param.py::test_stacked[~ -0]
test_param.py::test_stacked[~ -59]
test_param.py::test_stacked[~ -245]

14 tests collected in 0.01s

$ python -m pytest -q
..............                                                           [100%]
14 passed in 0.01s

$ python -m pytest -q -k "an-hour"
.                                                                        [100%]
1 passed, 13 deselected in 0.00s

$ python -m pytest -q "test_param.py::test_total[items2-445]"
.                                                                        [100%]
1 passed in 0.00s

Three consequences fall out, and every one of them is a reason parametrize beats the loop. One: each row becomes its own item with its own generated id. pytest builds an id like test_total[items3-expected3] from the values, or from your ids= labels if you give them. So the report shows four results, not one, and a failure names the exact input that broke. Two: because they are separate items, each runs in its own call inside its own try/except, so case two failing does not stop case three. You get four independent verdicts and the full picture. Three: stacking two parametrize decorators produces the Cartesian product. A 3-row and a 2-row decorator generate 3 × 2 = 6 items, one per pair, which pytest builds by nesting the multiplication.

Now feel the contrast sharply against the loop-inside-a-test. That loop is one item. The first assert that fails raises and unwinds the entire function, so cases after it never execute, and a single red hides how many cases actually broke — one? all four? You cannot tell. Parametrize inverts every part of that: N items, N independent calls, N verdicts, and each failure self-labeling its input. Same four claims; two completely different execution structures, and only one of them tells you the truth.

That last line is the tell that ties this section back to collection. When pytest prints collected 6 items from a file with two test functions, it is not miscounting — parametrize is collection-time item generation. The same discovery machine that turned files into items also expands one function into a whole family of them. Discovery and multiplication are one mechanism, seen twice.

# ---------- tracks.py -- ch 19's module, with ch 13's guards ----------
"""Duration helpers for the jukebox."""


class PlaylistError(Exception):
    """Something is wrong with this playlist."""


LIBRARY = [("Blinding Lights", 200), ("Titanium", 245), ("Levels", 340)]


def as_clock(seconds):
    """Turn a whole number of seconds into m:ss."""
    minutes, secs = divmod(seconds, 60)
    return f"{minutes}:{secs:02d}"


def total(tracks):
    """Sum the seconds of an iterable of (title, seconds) pairs."""
    return sum(secs for _title, secs in tracks)


def add_song(shelf, title, seconds):
    """Append (title, seconds) to shelf. Refuse junk instead of storing it."""
    if seconds <= 0:
        raise ValueError(f"impossible length: {seconds}s")
    if any(t == title for t, _s in shelf):
        raise PlaylistError(f"{title!r} is already on the shelf")
    shelf.append((title, seconds))
    return shelf


# ---------- test_tracks.py ----------
import pytest

from tracks import LIBRARY, PlaylistError, add_song, as_clock, total


def test_total_sums_the_library():
    assert total(LIBRARY) == 785


def test_total_of_nothing_is_zero():
    assert total([]) == 0


@pytest.mark.parametrize("seconds,expected", [
    (0,    "0:00"),
    (59,   "0:59"),
    (60,   "1:00"),
    (200,  "3:20"),
    (245,  "4:05"),
    (3600, "60:00"),
])
def test_as_clock_formats(seconds, expected):
    assert as_clock(seconds) == expected


def test_add_song_appends_one_row():
    shelf = [("Blinding Lights", 200)]
    add_song(shelf, "Titanium", 245)
    assert shelf == [("Blinding Lights", 200), ("Titanium", 245)]


def test_add_song_refuses_a_duplicate():
    shelf = [("Titanium", 245)]
    with pytest.raises(PlaylistError, match="already on the shelf"):
        add_song(shelf, "Titanium", 245)
    assert shelf == [("Titanium", 245)]      # and it did NOT append


def test_add_song_refuses_an_impossible_length():
    with pytest.raises(ValueError) as excinfo:
        add_song([], "Titanium", -50)
    assert "impossible length: -50s" in str(excinfo.value)


# ---------- now seed ONE bug in tracks.py and run it again ----------
    shelf.append([title, seconds])   # BUG: a list, not a tuple

$ python -m pytest -v
============================= test session starts =============================
platform win32 -- Python 3.12.7, pytest-9.1.1, pluggy-1.6.0
rootdir: C:\tmp\ch24lang\lab
collecting ... collected 11 items

test_tracks.py::test_total_sums_the_library PASSED                       [  9%]
test_tracks.py::test_total_of_nothing_is_zero PASSED                     [ 18%]
test_tracks.py::test_as_clock_formats[0-0:00] PASSED                     [ 27%]
test_tracks.py::test_as_clock_formats[59-0:59] PASSED                    [ 36%]
test_tracks.py::test_as_clock_formats[60-1:00] PASSED                    [ 45%]
test_tracks.py::test_as_clock_formats[200-3:20] PASSED                   [ 54%]
test_tracks.py::test_as_clock_formats[245-4:05] PASSED                   [ 63%]
test_tracks.py::test_as_clock_formats[3600-60:00] PASSED                 [ 72%]
test_tracks.py::test_add_song_appends_one_row PASSED                     [ 81%]
test_tracks.py::test_add_song_refuses_a_duplicate PASSED                 [ 90%]
test_tracks.py::test_add_song_refuses_an_impossible_length PASSED        [100%]

============================= 11 passed in 0.01s ==============================

$ python -m pytest -q          # after seeding the bug
........F..                                                              [100%]
================================== FAILURES ===================================
________________________ test_add_song_appends_one_row ________________________

    def test_add_song_appends_one_row():
        shelf = [("Blinding Lights", 200)]
        add_song(shelf, "Titanium", 245)
>       assert shelf == [("Blinding Lights", 200), ("Titanium", 245)]
E       AssertionError: assert [('Blinding L...tanium', 245]] == [('Blinding L...tanium', 245)]
E
E         At index 1 diff: ['Titanium', 245] != ('Titanium', 245)
E         Use -v to get more diff

test_tracks.py:30: AssertionError
=========================== short test summary info ===========================
FAILED test_tracks.py::test_add_song_appends_one_row - AssertionError: assert...
1 failed, 10 passed in 0.05s

Now your suite is green — 47 passed — and you feel safe. That feeling is the most dangerous thing in this chapter, because it is largely unearned. Next: exactly what a green bar licenses you to believe. →

06What green proves — and what it doesn't

The suite is green. 47 passed. You feel safe, and that feeling is the most dangerous thing in this whole chapter, because it is largely unearned. Green feels like 'my code is correct.' It is not. It means, exactly, 'the specific claims I remembered to write still hold.' The gap between those two sentences is not a technicality. It is where every shipped bug lives. This section is the intellectual core of the chapter, the epistemics of testing. Its job is to install a permanent, accurate model of what a green bar does and does not license. That way you neither over-trust it (ship on green with three shallow tests) nor over-fear it (chase 100% coverage believing the number means correctness).

Formalize it. A test suite is a finite set of claims {c₁ … cₙ}. Green means every cᵢ evaluated true on this run. It says exactly nothing about any behavior you did not express as some cᵢ. This is Dijkstra's razor made concrete: testing shows the presence of bugs, never their absence. The input space of any nontrivial function is astronomically large, often infinitely so, and your tests sample a handful of points in it. Green proves the sampled points behave. The unsampled continuum, which is essentially all of it, stays uncharted. A test that never fails on the inputs you chose tells you nothing about the input you didn't.

Now dismantle the metric everyone reaches for to feel safe: coverage. Line coverage instruments the run — a tracer (via sys.settrace, or sys.monitoring in 3.12+) records which line numbers executed while your tests ran, and coverage percent is executed lines over total lines. Here is its exact, fatal failure: executed is not verified. A line runs during your test whether or not any assert checks what that line produced. Watch it fail live:

def total(items):
    return len(items)     # BUG: should be sum(items)

def test_total():
    result = total([200, 245])
    assert isinstance(result, int)   # passes! len returns an int too

# coverage: 100% — the buggy line ran.
# correctness: 0%  — result is 2, not 445, and no assert pinned the VALUE.

Read that and let it sting. The buggy return len(items) executed, and coverage proudly reports 100% of the function covered. The test passed green, because len returns an int just like sum does, and the only thing the assert pinned was the type. The wrong value — 2 instead of 445 — sailed straight through, on a fully-covered line, because no assertion ever looked at it. Coverage measured reach. It structurally cannot measure correctness, because correctness lives in the assertions, not in the execution. Push to branch coverage (did both sides of every if run) and it is a genuine refinement. But it does not touch this core gap: you can traverse every branch of a function and still assert nothing meaningful about what any of them produced.

you type
# ---------- press.py ----------
def press_cost(seconds):
    """Cost in cents to press one side. Sides over 300s cost double."""
    if seconds <= 0:
        return 0
    return 200 + seconds          # BUG: the over-300s doubling was never written


# ---------- test_press.py ----------
from press import press_cost


def test_it_runs():
    press_cost(340)                          # no assert at all


def test_returns_an_int():
    assert isinstance(press_cost(340), int)


def test_costs_something():
    assert press_cost(340) > 0


def test_nothing_is_free():
    assert press_cost(0) == 0


# ---------- the terminal ----------
$ python -m pytest -q
$ sed -i 's/200 + seconds/250 + seconds/' press.py     # plant a MUTANT
$ python -m pytest -q
$ sed -i 's/250 + seconds/200 + seconds/' press.py     # put the line back

# ---------- now add the one claim nobody wrote ----------
def test_a_long_side_costs_double():
    assert press_cost(340) == 1080

$ python -m pytest -q

you see
$ python -m pytest -q
....                                                                     [100%]
4 passed in 0.01s

$ sed -i 's/200 + seconds/250 + seconds/' press.py
$ python -m pytest -q
....                                                                     [100%]
4 passed in 0.01s

$ sed -i 's/250 + seconds/200 + seconds/' press.py
$ python -m pytest -q
....F                                                                    [100%]
================================== FAILURES ===================================
________________________ test_a_long_side_costs_double ________________________

    def test_a_long_side_costs_double():
>       assert press_cost(340) == 1080
E       assert 540 == 1080
E        +  where 540 = press_cost(340)

test_press.py:21: AssertionError
=========================== short test summary info ===========================
FAILED test_press.py::test_a_long_side_costs_double - assert 540 == 1080
1 failed, 4 passed in 0.03s

So what actually determines a suite's proof-strength? Two things, and neither of them is coverage. First, assertion quality: what each test pins down — value, type, side effect, raised error. An isinstance check pins almost nothing, while assert result == 445 pins the answer. Second, input sampling: which regions you probe, like boundaries, empties, negatives, and the off-by-one edges where bugs actually cluster. A function correct on the three medium inputs you tried and wrong on the empty list is green until the day you add the empty-list case. The bug was always there, and your sample just never touched it.

# ---------- skip.py, before ----------
def next_index(current, count):
    """Index of the next track. Wraps back to 0 at the end of the playlist."""
    return current + 1          # a bug lives here


# ---------- the four green tests ----------
$ python -m pytest -q
....                                                                     [100%]
4 passed in 0.01s


# The three answers.
#
# 1. Two behaviours: ADVANCE by one, and WRAP to 0 at the end. Three tests pin
#    advancing (weakly: one pins nothing, one pins only the type). Zero tests
#    pin wrapping -- and wrapping is the only behaviour the code gets wrong.
#
# 2. A three-track playlist has indices 0, 1, 2. The last valid index is 2, and
#    next_index(2, 3) must be 0. That single input is the entire hole.
#
# 3. `current + 1`    -- all four green. That IS the bug, sitting in plain sight.
#    `current + 1.0`  -- exactly ONE red: test_returns_an_int. The two value
#                        tests stay green, because 1.0 == 1 is True in Python.
#                        Only the isinstance check can feel this one.
#    `abs(current)+1` -- all four green. No test ever passes a negative, so the
#                        mutant is invisible. Two survivors out of three.


# ---------- the missing test ----------
def test_wraps_at_the_end():
    assert next_index(2, 3) == 0


$ python -m pytest -q
....F                                                                    [100%]
================================== FAILURES ===================================
____________________________ test_wraps_at_the_end ____________________________

    def test_wraps_at_the_end():
>       assert next_index(2, 3) == 0
E       assert 3 == 0
E        +  where 3 = next_index(2, 3)

test_skip.py:21: AssertionError
=========================== short test summary info ===========================
FAILED test_skip.py::test_wraps_at_the_end - assert 3 == 0
1 failed, 4 passed in 0.03s


# ---------- skip.py, after: one line ----------
def next_index(current, count):
    """Index of the next track. Wraps back to 0 at the end of the playlist."""
    return (current + 1) % count


$ python -m pytest -q
.....                                                                    [100%]
5 passed in 0.01s


# The two mutants, and what they prove.
#
#   (current + 1) % (count + 1)
#       1 failed, 4 passed   ->  assert 3 == 0  where 3 = next_index(2, 3)
#       Only the new test feels it. The four originals never look past index 1.
#
#   (current - 1) % count
#       3 failed, 2 passed   ->  assert 2 == 1  where 2 = next_index(0, 3)
#                                assert 0 == 2  where 0 = next_index(1, 3)
#                                assert 1 == 0  where 1 = next_index(2, 3)
#       Everything screams, and each red names the exact input it was given.
#
# Neither of those would have been caught before. That is the honest difference
# the fifth test bought -- not the fix, the ALARM standing around the fix.
#
# The one sentence to keep:
#   Green never told you the code was wrong, because nothing in the suite ever
#   asked it the one question the docstring had already promised an answer to.

If you want an honest measure of a suite — one that can't be fooled by lines that merely ran — the tool is mutation testing: deliberately corrupt the code (flip a + to a -, an < to <=) and rerun. If a test dies, that behavior was really being checked. If the suite stays green on the mutant, no test ever pinned that behavior, however high your coverage number. Mutation testing measures whether your assertions can feel a bug; coverage only measures whether your feet walked the floor. Keep the distinction for life.

If green only proves what you asserted, then the discipline is in when you write the assert. Next: red-green-refactor, and how a killed bug becomes a permanent antibody. →

07Red-green-refactor and the regression test

Two fears converge here, and one answer resolves both. The first: you are about to change working code to make it cleaner, and you are paralyzed. How do you restructure without breaking something you can't see? The second is deeper. You just spent three hours hunting a bug, found it, fixed one character, and moved on. And in six months someone will reintroduce that exact bug, and you will hunt it again from scratch. The three hours of knowledge you paid for evaporated the moment you deleted your debug prints. Both fears have the same cure: a test written at the right moment. Red-green-refactor gives you a rhythm that makes refactoring safe by construction, and the regression test turns each bug you kill into a permanent antibody. The truth to land: a bug's most valuable byproduct is not the fix. It is the test that guarantees the bug never returns.

Red-green-refactor is a discipline that cashes in the isolation and re-runnability the earlier sections built. The loop, mechanically:

# 1. RED — write the test FIRST, for behavior that doesn't exist yet, and RUN it.
def test_empty():
    assert total([]) == 0       # fails today — total([]) raises

# 2. GREEN — write the MINIMUM code to pass. The test is now your spec.
def total(items):
    return sum(items)             # sum([]) is 0 — green

# 3. REFACTOR — change structure freely; the green test is a tripwire.

RED is the step people skip, and skipping it is a quiet disaster. You write the test before the code and run it to watch it fail. Why? Because a test you never saw fail might be asserting nothing at all (recall the empty def test_nothing(): pass from section 1, and the coverage-lies trap from section 6). Watching it go red proves the test can fail, which proves it actually exercises the thing you think it does. GREEN is writing the least code that passes, and the test is now your executable definition of done. REFACTOR is the payoff. You change the implementation freely — sum() to a hand-rolled loop, one function into three — and the green test is a tripwire that fires the instant you break observable behavior. This is the whole reason 'refactor' is a real word and not a synonym for 'rewrite and hope'. Refactoring means changing structure without changing behavior, and 'without changing behavior' is only a checkable claim if behavior is pinned by tests.

Now the highest-value instance of RED: the regression test. A bug gets reported — total() crashes on the empty list. Before you fix it, you write a test that reproduces it: assert total([]) == 0, which is red right now, because of the bug. Then you fix the bug, and the test turns green. And here is the part that pays forever. That test now lives in the suite permanently, and its single job is to fail loudly if this exact bug ever comes back — through a careless edit, a bad merge, a refactor gone wrong. You paid three hours to understand the bug once. The regression test banks that understanding so no one ever has to pay it again.

# ---------- ask.py ----------
# chapter 13's robust input, with the loop stripped off so it is testable
def to_int(text, low=None, high=None):
    """Parse text as a whole number inside [low, high]. Refuse anything else."""
    try:
        value = int(text)
    except ValueError as e:
        raise ValueError(f"not a whole number: {text!r}") from e
    if low is not None and value < low:
        raise ValueError(f"{value} is below the minimum {low}")
    if high is not None and value > high:
        raise ValueError(f"{value} is above the maximum {high}")
    return value


# ---------- test_ask.py ----------
import pytest

from ask import to_int


def test_a_good_string_comes_back_as_an_int():
    assert to_int("245", low=1, high=600) == 245


def test_junk_names_what_it_saw():
    with pytest.raises(ValueError, match=r"not a whole number: 'abc'"):
        to_int("abc")


def test_below_the_minimum():
    with pytest.raises(ValueError, match=r"below the minimum 1"):
        to_int("0", low=1, high=600)


def test_above_the_maximum():
    with pytest.raises(ValueError, match=r"above the maximum 600"):
        to_int("601", low=1, high=600)


def test_the_original_error_is_kept_as_the_cause():
    with pytest.raises(ValueError) as excinfo:
        to_int("4:05")
    assert isinstance(excinfo.value.__cause__, ValueError)
    assert "invalid literal" in str(excinfo.value.__cause__)


def test_the_boundaries_are_inclusive():
    assert to_int("1", low=1, high=600) == 1
    assert to_int("600", low=1, high=600) == 600


$ python -m pytest -q
......                                                                   [100%]
6 passed in 0.01s


# ---------- the trap: a to_int that validates NOTHING ----------
def to_int(text, low=None, high=None):
    value = int(text)                       # no translation, no bounds at all
    return value


import pytest
from ask import to_int


def test_bare_raises_is_fooled():
    with pytest.raises(ValueError):         # green -- but for the WRONG reason
        to_int("abc")


def test_match_is_not_fooled():
    with pytest.raises(ValueError, match=r"not a whole number"):
        to_int("abc")


$ python -m pytest -q
.F                                                                       [100%]
================================== FAILURES ===================================
__________________________ test_match_is_not_fooled ___________________________

    def test_match_is_not_fooled():
>       with pytest.raises(ValueError, match=r"not a whole number"):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AssertionError: Regex pattern did not match.
E         Expected regex: 'not a whole number'
E         Actual message: "invalid literal for int() with base 10: 'abc'"

test_trap.py:12: AssertionError
=========================== short test summary info ===========================
FAILED test_trap.py::test_match_is_not_fooled - AssertionError: Regex pattern...
1 failed, 1 passed in 0.03s


# The rule, and its price.
#
#   A bare pytest.raises(ValueError) asserts only that SOMETHING went wrong. It
#   is green when int() blew up on its own and your validator never ran a line.
#   match= asserts WHICH thing went wrong, so the test can tell your error apart
#   from the accident underneath it.
#
#   The price: match= is a regex, so a reworded message breaks the test. That is
#   the trade you are making -- the message becomes part of the contract. Match
#   on the stable half of the sentence ("not a whole number"), never on the
#   whole line, and never on a formatted number you might reformat next month.
#
# The one sentence to keep:
#   Testing that it raises is easy. Testing that it raises for YOUR reason is
#   the test.

Step back and see what a mature suite actually is. Section 6 killed the idea that it proves correctness — it doesn't, and it can't. So what is it? It is institutional memory. Every regression test is a dated entry in a diary: on this day, this input broke this function. Here is the claim that ensures it never breaks the same way again. A suite is not a proof of correctness, but the accumulated scar tissue of every bug the code has survived. And that reframes an old intuition. Old code with a real suite is safer to change than new code without one, precisely because the old code's suite remembers hundreds of specific ways it once broke, and stands guard over every one of them.

# ---------- test_smells.py -- as given, all four green ----------
from tracks import add_song, as_clock, total

SHELF = []                                  # one list, shared by the whole file


def test_add_song():                        # SMELL 1
    shelf = []
    add_song(shelf, "Titanium", 245)


def test_the_playlist():                    # SMELL 2
    shelf = []
    add_song(shelf, "Blinding Lights", 200)
    assert len(shelf) == 1
    add_song(shelf, "Titanium", 245)
    assert total(shelf) == 445
    assert as_clock(total(shelf)) == "7:25"


def test_a_adds_one():                      # SMELL 3
    add_song(SHELF, "Blinding Lights", 200)
    assert len(SHELF) == 1


def test_b_adds_another():                  # SMELL 3
    add_song(SHELF, "Titanium", 245)
    assert total(SHELF) == 445


$ python -m pytest -q
....                                                                     [100%]
4 passed in 0.01s


# ============ SMELL 1: ASSERTS NOTHING (a smoke test wearing a test's name) ==
# It proves the call did not raise. It proves nothing about what the call did.
# Proof -- make add_song return without appending, then run both versions:

    def test_add_song():                   # asserts nothing
        shelf = []
        add_song(shelf, "Titanium", 245)

    def test_add_song_appends_the_row():   # pins the result
        shelf = []
        add_song(shelf, "Titanium", 245)
        assert shelf == [("Titanium", 245)]

$ python -m pytest -q
.F                                                                       [100%]
________________________ test_add_song_appends_the_row ________________________

    def test_add_song_appends_the_row():   # pins the result
        shelf = []
        add_song(shelf, "Titanium", 245)
>       assert shelf == [("Titanium", 245)]
E       AssertionError: assert [] == [('Titanium', 245)]
E
E         Right contains one more item: ('Titanium', 245)

# The dot is the assertless test, still green over a function that does nothing.
# FIX: assert the result, not the absence of a crash.


# ============ SMELL 2: TWO (here THREE) BEHAVIOURS IN ONE TEST ==============
# Proof -- plant TWO bugs (total -> len, as_clock -> no zero-pad) and run it:

$ python -m pytest -q
F                                                                        [100%]
______________________________ test_the_playlist ______________________________

    def test_the_playlist():
        shelf = []
        add_song(shelf, "Blinding Lights", 200)
        assert len(shelf) == 1
        add_song(shelf, "Titanium", 245)
>       assert total(shelf) == 445          # fails HERE
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AssertionError: assert 2 == 445
E        +  where 2 = total([('Blinding Lights', 200), ('Titanium', 245)])

1 failed in 0.03s

# Two bugs in the file, ONE reported. The assert on as_clock never ran, so the
# formatting bug is invisible until you fix the first one and run again. And the
# name in the report -- 'test_the_playlist' -- names no behaviour at all.
# FIX: one behaviour per test, and the test's name IS the failure report.


# ============ SMELL 3: ORDER-DEPENDENT (shared mutable state) ===============
# Proof -- change no code. Run the SAME two tests in the other order:

$ python -m pytest -q "test_smells.py::test_b_adds_another" \
                      "test_smells.py::test_a_adds_one"
FF                                                                       [100%]
_____________________________ test_b_adds_another _____________________________
>       assert total(SHELF) == 445
E       AssertionError: assert 245 == 445
E        +  where 245 = total([('Titanium', 245)])
_______________________________ test_a_adds_one _______________________________
>       assert len(SHELF) == 1
E       AssertionError: assert 2 == 1
E        +  where 2 = len([('Titanium', 245), ('Blinding Lights', 200)])

2 failed in 0.03s

# Both of them. The pair was never two tests -- it was one script split in half,
# and file order was load-bearing. This is the bug that only reproduces on CI.
# FIX: a function-scoped fixture. Fresh object per test, by construction.


# ---------- test_fixed.py -- the rewrite ----------
import pytest

from tracks import add_song, as_clock, total


@pytest.fixture
def shelf():
    return []                               # fresh list per test -- no sharing


def test_add_song_appends_the_row(shelf):           # FIX 1: pin the result
    add_song(shelf, "Titanium", 245)
    assert shelf == [("Titanium", 245)]


def test_add_song_keeps_the_order(shelf):           # FIX 2: one behaviour each
    add_song(shelf, "Blinding Lights", 200)
    add_song(shelf, "Titanium", 245)
    assert shelf == [("Blinding Lights", 200), ("Titanium", 245)]


def test_total_sums_two_rows():
    assert total([("Blinding Lights", 200), ("Titanium", 245)]) == 445


def test_as_clock_formats_the_total():
    assert as_clock(445) == "7:25"


$ python -m pytest test_fixed.py -q
....                                                                     [100%]
4 passed in 0.01s

$ python -m pytest -q "test_fixed.py::test_add_song_keeps_the_order" \
                      "test_fixed.py::test_add_song_appends_the_row"
..                                                                       [100%]
2 passed in 0.00s

# Same two tests, other order, same verdict. That is what isolation looks like
# from the outside, and it cost one fixture.
#
# The one sentence to keep:
#   A test that asserts nothing cannot fail, a test that asserts two things
#   hides one of them, and a test that shares state asserts something about the
#   test before it.

And that closes the whole chapter's circle. The regression test is an executable claim (§1), discovered by collection (§2), whose failure talks through the rewrite (§3). It is often fed by a fixture (§4) and generalized by parametrize (§5). And it is written by someone who knows, without illusion, that it proves only what it asserts (§6). Green never meant 'correct.' It meant 'the claims I was disciplined enough to write are still true.' Make those claims sharp, sample the edges where bugs hide, and turn every bug you kill into a claim that outlives you. That is the whole craft.

You can now keep a claim about your code alive forever — and you know exactly what that claim does and doesn't prove. Next volume-thread stop: when the thing crossing the boundary isn't a value you can assert on, but bytes on a wire. →