15Files are syscalls — the file descriptor

In Volume 1 we lived entirely inside one running process. Every object sat at a real address in RAM, every name pointed straight at it, and the whole world was one warm, private room. In this chapter we cut a door in that wall, and here's the plan. A program that does anything useful has to reach past itself — to the disk, to other encodings, to other people's code, to future-you. And every one of those boundaries obeys a single law: what crosses a process boundary is always bytes, never objects. We start at the oldest boundary of all, the one between memory that forgets and memory that remembers. The whole way through, we keep asking the one thing that actually matters — if a file isn't a Python object, then what is it, and what really happens the instant you type open('song.txt')? By the end you'll see the answer with your own eyes: a "file" is a kernel thing you're only ever allowed to touch through a tiny integer. And open, read, write, close are four traps into the operating system, each wearing a friendly object as a disguise.

01Why disk at all: the volatile / non-volatile split

Let's start with a small cruelty you've lived with since your very first program. You probably never said it out loud: every script you have ever written forgot everything the instant it ended. That list you spent an hour building, a running total, a game's high score — all gone the moment the process exits, as if it had never existed. We tend to shrug it off. "That's just how programs work." But watch closely, because that shrug hides a real physical fact. Until you feel why memory forgets, open() looks like an arbitrary API. It is really something else: the only door out of a burning building.

So look at where your data actually lives. In Volume 1 every object sat in DRAM — dynamic random-access memory, the main memory of your machine. Here's what DRAM actually is under the name. It stores each single bit as a puff of electric charge on a microscopic capacitor, gated by one transistor. Engineers call it the 1T1C cell: one transistor and one capacitor per bit. And here's the fatal part. That capacitor is tiny, a few femtofarads, and it leaks. Charge drains through the transistor's imperfect off-state in a matter of milliseconds. Leave a stored 1 alone and it decays into an unreadable smear in well under a tenth of a second.

So DRAM cheats to stay alive. The memory controller marches through every row of every chip roughly every 64 milliseconds and performs a refresh. It reads each row and immediately writes it back, re-pouring charge into buckets that are always leaking. Your "static" variable is in fact a frantically re-topped-up puddle. This is not a metaphor. It is happening billions of times a second in the machine reading this sentence. Now cut the power. Refresh stops within a single cycle, and every capacitor drains to the noise floor. The bits are not "lost by the operating system" or "cleared on shutdown." They are physically gone, because the physics that held them needed a power supply, and that supply is gone. This is what the word volatile means, all the way down.

That different hardware exists, and it works by a completely different trick. It stores state in a form that survives with zero power. A NAND flash cell traps a clump of electrons on a floating gate — a scrap of conductor completely sealed behind an oxide insulator. Those trapped electrons shift the transistor's threshold voltage, and the oxide wall holds them in place for years with nothing plugged in. A spinning hard disk goes older still. It stores each bit as the magnetization direction of a grain of ferromagnetic material on a platter. A magnetized grain simply stays magnetized when the lights go out. Flash and disk are non-volatile. Pull the plug and they lose nothing.

This is the axiom the entire chapter — the entire volume — rests on. There is a hard line running through your computer's hardware. On one side sits fast volatile silicon that forgets the instant it is unpowered, and on the other, slower non-volatile media that remember without power. A "file" is nothing more grand than a byte that has crossed to the remembering side. That is the whole concept. The reason open() exists, the reason files exist as an idea, is simple: some hardware can hold a bit without being fed, and RAM cannot.

You can feel the split with three lines of code: write a program that counts, and run it twice:

# counter.py — run me twice. I will always say 1.
count = 0
count = count + 1
print(count)            # 1 ... and 1 again on the next run, and forever

It restarts at zero every time because count lives in DRAM, and DRAM forgot it the microsecond the process died. Now change one thing. Read the previous value from a file at the top, and write the new one back at the bottom. Suddenly the count climbs across separate runs: 1, 2, 3, surviving the death of the program each time. Nothing about the arithmetic changed. What changed is that one number physically left volatile silicon and landed on the remembering side. That crossing is what the rest of this chapter takes apart, mechanism by mechanism.

So a byte has to reach non-volatile hardware. But your Python process is a plain user program with no wire to the disk — if it tried to command the drive directly, the CPU would fault. So how does one innocent line, open('song.txt'), reach a device you are forbidden to touch? →

02open() is a trap into the kernel

You now know a byte has to reach non-volatile hardware to survive. But sit with an uncomfortable fact: your Python process is not allowed to touch that hardware. Your code cannot issue a command to the SATA or NVMe controller, poke the disk's registers, or nudge the spinning platter. Suppose it tried, and the CPU, while running your bytecode, attempted the privileged instruction that talks to a device controller. The processor would raise a fault, and the operating system would kill your program on the spot. So we have a paradox. One line, open('song.txt'), clearly does reach the disk. How, if your code is forbidden from doing so directly?

The answer lives in the CPU itself, in a mechanism called privilege rings. The processor is always running at some privilege level. Your program, like every ordinary application, runs in ring 3, the unprivileged, fenced-in level. The operating system kernel runs in ring 0, with full authority over the machine. Certain instructions are privileged: talking to device controllers, remapping the page tables that define memory, touching I/O ports. Attempt any of them from ring 3 and the hardware itself slams the door. You get a fault, not a warning. This is enforced by the silicon, not by politeness. By design, a user process has no instruction that reaches the disk.

So how do you ever get anything done? Through the one sanctioned door in the wall: a syscall. A syscall is a special CPU instruction — literally syscall on x86-64, svc on ARM64 — that does something no ordinary instruction can. It atomically switches the CPU into ring 0 and jumps to a single fixed entry point inside the kernel. The hardware was pre-told about that entry point at boot. You don't get to choose where in the kernel you land. You get exactly one guarded gate. Which service you're asking for is selected by a number you place in a register beforehand. On Linux x86-64, register rax holds the syscall number (openat is number 257). This controlled, one-way crossing is called a trap. The program traps into the kernel, the kernel does the privileged work on your behalf, and control traps back out to ring 3.

Follow one open through the gate. Your ring-3 code executes the syscall instruction, and the CPU flips to ring 0 and lands at the kernel entry point, now running with full privilege. It walks the filesystem to find song.txt, talks through the block-device driver to the actual disk controller, and sets up its bookkeeping for this newly opened file. Then it hands back a single small non-negative integer. The CPU flips back to ring 3, and your program resumes with that integer in hand. That integer is the file descriptor — the kernel's receipt, the handle you will use for every later read and write. It is not the file, but a claim ticket the kernel gave you.

CPython's friendly open() is many layers of Python and C sitting on top of exactly this syscall. To see the raw thing with nothing dressed on it, use os.open. That thin wrapper does little more than the trap and hands you the number back:

import os
fd = os.open("song.txt", os.O_RDONLY)
print(fd)          # 3 — a bare integer. Not an object. The kernel's receipt.

There it is: 3. Not a file object, not a path, not a stream — an int. That is the correction this section exists to make. open() is not "Python opening a file." It is Python asking the kernel, across a hardware-enforced mode switch, to open a file on Python's behalf and to hand back a handle. The friendly f = open(...) object you're used to has that same integer buried inside it; call f.fileno() and it will show you the very number the kernel returned. Everything else is wrapping paper around a receipt.

you type
# modes.py -- one call, six contracts. The mode letter is the whole promise.
from pathlib import Path

def show(path):                                  # read helper, written the safe way
    with open(path, "r", encoding="utf-8") as f:
        return repr(f.read())

Path("shelf.txt").unlink(missing_ok=True)        # start from an empty shelf

# 'w' - create if absent, TRUNCATE to zero if present, then write
with open("shelf.txt", "w", encoding="utf-8") as f:
    f.write("Blinding Lights\n")
print("after 'w' :", show("shelf.txt"))

# 'a' - append: every write lands at the end, nothing already there is touched
with open("shelf.txt", "a", encoding="utf-8") as f:
    f.write("Titanium\n")
print("after 'a' :", show("shelf.txt"))

# 'w' a second time - the truncation everyone forgets, felt once
with open("shelf.txt", "w", encoding="utf-8") as f:
    f.write("Levels\n")
print("after 'w' :", show("shelf.txt"), "<- the first two songs are gone")

# 'x' - exclusive create: refuses to clobber a file that already exists
try:
    with open("shelf.txt", "x", encoding="utf-8") as f:
        f.write("never gets here")
except FileExistsError as e:
    print("mode 'x'  :", type(e).__name__, "- it will not overwrite an existing file")

# 'r' - read, and the file must already exist
try:
    with open("missing.txt", "r", encoding="utf-8") as f:
        f.read()
except FileNotFoundError as e:
    print("mode 'r'  :", type(e).__name__, "- errno", e.errno)

# 'b' suffix - no codec, no newline translation. Bytes in, bytes out.
with open("cover.bin", "wb") as f:
    n = f.write(b"ID3\x03\x00")
with open("cover.bin", "rb") as f:
    print("mode 'wb' :", n, "bytes written | mode 'rb' :", f.read())

# '+' suffix - read AND write on one descriptor, without truncating
with open("shelf.txt", "r+", encoding="utf-8") as f:
    print("mode 'r+' :", repr(f.read()), "read first...")
    f.write("bad guy\n")                         # ...then write from where reading stopped
print("after 'r+':", show("shelf.txt"))

you see
after 'w' : 'Blinding Lights\n'
after 'a' : 'Blinding Lights\nTitanium\n'
after 'w' : 'Levels\n' <- the first two songs are gone
mode 'x'  : FileExistsError - it will not overwrite an existing file
mode 'r'  : FileNotFoundError - errno 2
mode 'wb' : 5 bytes written | mode 'rb' : b'ID3\x03\x00'
mode 'r+' : 'Levels\n' read first...
after 'r+': 'Levels\nbad guy\n'

The kernel handed you the integer 3 — never 0, 1, or 2. That's not luck and it's not random. Why does counting start at 3, and where does the number even live? →

you type
# paths.py -- a path is an object, not a string you glue together
from pathlib import Path
import os

songs = Path("data") / "songs.txt"               # the / operator joins, OS-correctly
print("path      :", songs)
print("parts     :", songs.parent, "|", songs.name, "|", songs.stem, "|", songs.suffix)

songs.parent.mkdir(parents=True, exist_ok=True)  # make data/ if it isn't there yet
print("exists?   :", songs.exists(), "before writing")

songs.write_text("Blinding Lights\nTitanium\n", encoding="utf-8")   # open+write+close
print("exists?   :", songs.exists(), "| is_file:", songs.is_file())
print("read_text :", repr(songs.read_text(encoding="utf-8")))

(songs.parent / "cover.bin").write_bytes(b"ID3\x03\x00")            # the bytes twin
print("read_bytes:", (songs.parent / "cover.bin").read_bytes())

for p in sorted(songs.parent.iterdir()):         # what is in this directory
    print("  entry   :", p.name, "| suffix", p.suffix or "(none)")
print("glob *.txt:", [p.name for p in sorted(songs.parent.glob("*.txt"))])

# the old spelling, for when you read someone else's code
legacy = os.path.join("data", "songs.txt")
print("os.path   :", legacy, "| same file:", os.path.exists(legacy))

with open(songs, "r", encoding="utf-8") as f:    # open() accepts a Path directly
    print("interop   :", f.readline().strip(), "<- open() takes a Path, no str() needed")

you see
path      : data\songs.txt
parts     : data | songs.txt | songs | .txt
exists?   : False before writing
exists?   : True | is_file: True
read_text : 'Blinding Lights\nTitanium\n'
read_bytes: b'ID3\x03\x00'
  entry   : cover.bin | suffix .bin
  entry   : songs.txt | suffix .txt
glob *.txt: ['songs.txt']
os.path   : data\songs.txt | same file: True
interop   : Blinding Lights <- open() takes a Path, no str() needed

03The fd table, and why 0, 1, 2 are already taken

Open your first file and you get descriptor 3. Open one in a brand-new program tomorrow, and it's still 3. Never 0, never 1, never 2. That is astonishingly consistent for something that looks arbitrary. Consistency is always a clue that a structure is doing the choosing, and there is one here. And if you don't understand the array these integers index into, then leaks, reuse, and "why did I just read the wrong file?" will confuse you for the rest of your career. So let's make the descriptor concrete.

Every process has its own private file-descriptor table — an array living inside the kernel, hanging off the kernel's bookkeeping struct for your process. It is indexed by the descriptor integer. Entry i doesn't hold the file itself. It holds a pointer to a kernel open-file description — a small record of the current read/write offset, the flags you opened with, and a link to the underlying file on disk (its inode). The descriptor you hold in Python is just the index. All the real state lives on the kernel's side of the wall. This is why the descriptor is a mere integer. It is an array subscript, and the array isn't yours to see.

Now the mystery of 0, 1, 2. When the shell (or whatever launched you) starts your process, it does not hand you an empty table. It pre-wires the first three slots. Index 0 is standard input, index 1 is standard output, and index 2 is standard error. Each already points at your terminal, or wherever they were redirected. This is why print reaches your screen although you never opened anything. print ultimately writes to descriptor 1, which was bound to your terminal before your first line ran. You inherited a working output before you did anything at all.

And here is the rule that ties it together: open always installs the new file at the lowest free index in the table. With 0, 1, and 2 already occupied, your first open can only land at 3. The next at 4. The next at 5. There is no randomness — it is just "smallest empty slot," every time.

import os, sys
print(sys.stdin.fileno(), sys.stdout.fileno(), sys.stderr.fileno())  # 0 1 2
a = os.open("song.txt", os.O_RDONLY)
b = os.open("song.txt", os.O_RDONLY)
print(a, b)      # 3 4 — the two lowest free slots, in order

Two consequences fall straight out of "it's a private, finite array." First, the table is per-process. Your descriptor 3 and another program's descriptor 3 index two entirely different arrays, pointing at two entirely different files. Descriptors are not global names. A 3 only means anything inside the process that owns the table. Second, the array is finite. Its capacity is capped by a limit named RLIMIT_NOFILE, the thing your shell's ulimit -n reports, often 1024 by default. You may hold only that many open descriptors at once.

That ceiling is not theoretical. Open files in a loop and never close them, and you watch the descriptors climb: 3, 4, 5, …, 1021, 1022, 1023. Then the program dies with OSError: [Errno 24] Too many open files. That is the finite table felt as a crash, and a leaked descriptor is a slot you claimed and never gave back.

The flip side of "lowest free index" is reuse. Close descriptor 3 and its slot goes empty again, so the very next open hands you 3 back, not 5. This is the source of one of the nastiest bugs in systems programming. You close a file, keep a copy of the old number 3 lying around, and later something else opens a different file that also gets 3. Now your stale 3 silently reads the wrong file. The kernel is perfectly happy to comply, because to it your 3 is a valid index into a valid slot. The number was reclaimed. Your memory of what it meant was not.

>>> from pathlib import Path
>>> Path("song.txt").write_text("My Favorite Song", encoding="utf-8")
>>> a = open("song.txt", encoding="utf-8")   # the same file...
>>> b = open("song.txt", encoding="utf-8")   # ...opened a second time
>>> a.read()                                 # a drains the whole file
>>> b.read()                                 # and b? untouched by that

16
'My Favorite Song'
'My Favorite Song'

You hold descriptor 3. Now you actually want the file's bytes inside a Python variable — but your memory and the disk sit on opposite sides of the ring wall. How do bytes physically cross, and does the kernel have any idea what they mean? →

04read and write copy bytes across the ring boundary

You hold descriptor 3. You want two things: the file's contents pulled into a Python variable, and your own bytes pushed out onto the disk. But recall where these two live. Your process's memory sits in ring-3 user space; the disk and the kernel's buffers sit behind the ring-0 wall. They are different memory that the CPU deliberately keeps apart. So the real question of this section is physical: when data moves between your variable and the file, what actually happens to it? And a quieter unease sits underneath — when you write the string "My Favorite Song", does the kernel understand it's a song title?

Both movements are syscalls, each a trap into ring 0 exactly like open was. read(fd, n) asks the kernel to copy up to n bytes from its buffers for that file into a buffer in your process's address space. write(fd, buf) copies the other direction. It moves bytes from your memory into the kernel's buffers, and the kernel later flushes them to the physical device. Notice the load-bearing word in both sentences: copy. Your user pages and the kernel's buffers are separate memory the CPU won't let mix. So the kernel must physically move the bytes across the boundary — a real memcpy, plus the cost of the mode switch. That double cost is a trap and a copy on every call. It is the entire reason I/O is expensive, and the entire reason we buffer instead of doing a syscall per byte.

you type
# reading.py -- one file, four ways in, and the position that moves under you
from pathlib import Path

Path("playlist.txt").write_text(
    "Blinding Lights\nTitanium\nLevels\nbad guy\n", encoding="utf-8")

# 1 - .read() : the whole file as one str. Fine for small files.
with open("playlist.txt", "r", encoding="utf-8") as f:
    whole = f.read()
print("read()      :", repr(whole))
print("             ", len(whole), "characters, newlines and all")

# 2 - .read(n) : at most n characters, from wherever the position is now
with open("playlist.txt", "r", encoding="utf-8") as f:
    print("read(8)     :", repr(f.read(8)))
    print("read(8) x2  :", repr(f.read(8)), "<- it carried on from where it stopped")

# 3 - .readline() / .readlines() : the newline comes along for the ride
with open("playlist.txt", "r", encoding="utf-8") as f:
    print("readline()  :", repr(f.readline()))
    print("readlines() :", f.readlines(), "<- the rest, as a list")

# 4 - iterate the file object : THE idiomatic loop. One line at a time, lazily.
with open("playlist.txt", "r", encoding="utf-8") as f:
    for n, line in enumerate(f, start=1):
        clean = line.rstrip("\n")              # the newline is IN the string until you strip it
        print("  line", n, "  :", repr(line), "->", repr(clean))

# the trap: the position sits at the end now, so a second read gives you nothing
with open("playlist.txt", "r", encoding="utf-8") as f:
    first = f.read()
    second = f.read()
print("second read :", repr(second), "<- empty. The file is fine; you are at the end.")

with open("playlist.txt", "r", encoding="utf-8") as f:
    f.read()
    f.seek(0)                                  # rewind the position by hand
    print("after seek(0):", repr(f.read(15)))

# .splitlines() : the same lines with the newlines already gone
print("splitlines():", whole.splitlines())

you see
read()      : 'Blinding Lights\nTitanium\nLevels\nbad guy\n'
              40 characters, newlines and all
read(8)     : 'Blinding'
read(8) x2  : ' Lights\n' <- it carried on from where it stopped
readline()  : 'Blinding Lights\n'
readlines() : ['Titanium\n', 'Levels\n', 'bad guy\n'] <- the rest, as a list
  line 1   : 'Blinding Lights\n' -> 'Blinding Lights'
  line 2   : 'Titanium\n' -> 'Titanium'
  line 3   : 'Levels\n' -> 'Levels'
  line 4   : 'bad guy\n' -> 'bad guy'
second read : '' <- empty. The file is fine; you are at the end.
after seek(0): 'Blinding Lights'
splitlines(): ['Blinding Lights', 'Titanium', 'Levels', 'bad guy']

Now the crucial part, the one that reframes what a file is. The payload is always raw bytes — an array of integers from 0 to 255, nothing more. The kernel has zero notion that your bytes spell a song title, or encode a JSON object, or form a JPEG. When you write b"My Favorite Song", the kernel sees the sequence 77, 121, 32, 70, 97, 118, … and faithfully moves those numbers. It does not see "My Fav…". It cannot. Meaning is not something the kernel stores or checks. Meaning is imposed entirely by your program, above the wall. The file on disk is just a byte sequence. The fact that it "is" a song title or a picture is an agreement between the program that wrote it and the program that reads it. The kernel is a neutral courier of opaque numbers.

you type
# writing.py -- four ways out, and the newline nobody adds for you
from pathlib import Path

Path("out.txt").unlink(missing_ok=True)

def show(path):
    with open(path, "r", encoding="utf-8") as f:
        return repr(f.read())

# 1 - .write(s) : returns a COUNT, and adds nothing you did not type
with open("out.txt", "w", encoding="utf-8") as f:
    n1 = f.write("Blinding Lights")
    n2 = f.write("Titanium")
print("write() ->", n1, n2, "characters |", show("out.txt"), "<- one run-on line")

# 2 - the fix: put the newline in yourself
with open("out.txt", "w", encoding="utf-8") as f:
    f.write("Blinding Lights\n")
    f.write("Titanium\n")
print("with \\n  ->", show("out.txt"))

# 3 - .writelines(seq) : writes them all, and STILL adds no newlines
rows = ["Levels", "bad guy"]
with open("out.txt", "w", encoding="utf-8") as f:
    f.writelines(rows)
print("writelines->", show("out.txt"), "<- the trap: no separators at all")

with open("out.txt", "w", encoding="utf-8") as f:
    f.writelines(line + "\n" for line in rows)     # you supply the newlines
print("writelines->", show("out.txt"))

# 4 - print(..., file=f) : the one call that DOES add the newline
with open("out.txt", "w", encoding="utf-8") as f:
    for title in ("Levels", "bad guy"):
        print(title, file=f)
print("print(file=)->", show("out.txt"))

# 5 - one join, one write: the fastest spelling for a list you already have
with open("out.txt", "w", encoding="utf-8") as f:
    f.write("\n".join(rows) + "\n")
print("join+write ->", show("out.txt"))

# 'w' TRUNCATES; 'a' APPENDS. The same two lines, two different files.
with open("out.txt", "w", encoding="utf-8") as f:
    f.write("Titanium\n")
print("'w' again  ->", show("out.txt"), "<- everything before it is gone")

with open("out.txt", "a", encoding="utf-8") as f:
    f.write("Levels\n")
print("'a'        ->", show("out.txt"), "<- added at the end instead")

you see
write() -> 15 8 characters | 'Blinding LightsTitanium' <- one run-on line
with \n  -> 'Blinding Lights\nTitanium\n'
writelines-> 'Levelsbad guy' <- the trap: no separators at all
writelines-> 'Levels\nbad guy\n'
print(file=)-> 'Levels\nbad guy\n'
join+write -> 'Levels\nbad guy\n'
'w' again  -> 'Titanium\n' <- everything before it is gone
'a'        -> 'Titanium\nLevels\n' <- added at the end instead

There is one more truth the friendly API hides, and it bites everyone eventually. These syscalls return a count. write returns how many bytes the kernel actually accepted; read returns how many it actually delivered — and that can be fewer than you asked for. Ask read for 4096 bytes near the end of a file and you'll get only the handful that remain. Read from a pipe or a socket and you may get a trickle. This is called a short read (and its sibling, the short write), and assuming a single call moved everything is a classic, painful bug.

import os
fd = os.open("song.txt", os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644)
n = os.write(fd, b"My Favorite Song")
print(n)              # 16 — the count of bytes accepted, not True, not None
os.close(fd)

fd = os.open("song.txt", os.O_RDONLY)
print(os.read(fd, 4))    # b'My F' — only 4 bytes; a deliberate short read

Look at what write handed back: 16, not True. It is telling you it moved sixteen bytes. You're expected to check that, because in the general case it might have moved fewer, and you'd need to loop and send the rest. And read(fd, 4) gave you b'My F' — four raw bytes, prefixed b for bytes, because that is genuinely all the kernel deals in. Your program is the thing that decides those four numbers are letters.

You saw a leak crash a program by never closing. So closing matters — but you can't just add a close() at the end and relax, because what if the line before it raises? →

# write_then_read.py -- one file out, the same file back in three ways
from pathlib import Path

TRACKS = ["Blinding Lights", "Titanium", "Levels", "bad guy"]
PATH = Path("playlist.txt")

# ---------- out: one with-block, one line per track ----------
with open(PATH, "w", encoding="utf-8") as f:
    for title in TRACKS:
        f.write(title + "\n")                  # you type the newline; write never adds it
print("wrote", len(TRACKS), "tracks to", PATH.name)
print()

# ---------- way 1: .read() -- the whole file as ONE string ----------
with open(PATH, "r", encoding="utf-8") as f:
    blob = f.read()
print("1) read()       ->", repr(blob))
print("   type/len     ->", type(blob).__name__, len(blob), "characters")
print()

# ---------- way 2: .readlines() -- a LIST of strings, newlines kept ----------
with open(PATH, "r", encoding="utf-8") as f:
    lines = f.readlines()
print("2) readlines()  ->", lines)
print("   stripped     ->", [ln.rstrip("\n") for ln in lines])
print()

# ---------- way 3: iterate the file object -- the idiomatic loop ----------
print("3) for line in f:")
with open(PATH, "r", encoding="utf-8") as f:
    for n, line in enumerate(f, start=1):
        print(f"     {n}. {line.rstrip()}")
print()

# the file object is closed on the way out of every block above
print("closed after the with-block:", f.closed)
print("same bytes, three shapes:", repr(blob) == repr("".join(lines)))

wrote 4 tracks to playlist.txt

1) read()       -> 'Blinding Lights\nTitanium\nLevels\nbad guy\n'
   type/len     -> str 40 characters

2) readlines()  -> ['Blinding Lights\n', 'Titanium\n', 'Levels\n', 'bad guy\n']
   stripped     -> ['Blinding Lights', 'Titanium', 'Levels', 'bad guy']

3) for line in f:
     1. Blinding Lights
     2. Titanium
     3. Levels
     4. bad guy

closed after the with-block: True
same bytes, three shapes: True

05close() and the context manager that guarantees it

The last section left you with a warning felt as a crash. Leak descriptors and you eventually slam into the finite table and die, so closing matters. The naive fix is obvious: put a close() at the end and move on. But that fix has a hole you can drive a truck through, and every teacher who ever nagged you about with open(...) was quietly pointing at it. What if a line between the open and the close raises an exception? Then execution leaps over your close(), and the descriptor leaks. Worse, the bytes you thought you'd saved may never reach the disk at all. This section is about making cleanup a guarantee instead of a hope.

First, what close(fd) actually does. It is itself a syscall. It returns your descriptor's slot to the process's fd table, freeing that index for reuse — exactly the reclamation we saw in section 3. It also tells the kernel you're finished with this open-file description. So the kernel can drop its reference, and when the last reference falls, release the underlying bookkeeping. Two separate things depend on close running: the finite descriptor table (unclosed descriptors leak until ulimit stops you) and durability. That second one is subtle. Your writes usually don't go straight to the platter. They sit in the kernel's buffers, and above that, in Python's own userspace buffer. Closing flushes Python's buffer down through the write syscall and hands clean state to the kernel. Skip the close and your final writes may still be sitting in a buffer that dies with the process.

Now the hole — consider the honest, well-meaning version:

f = open("song.txt", "w")
f.write("My Favorite Song")
risky()               # if this raises, control jumps AWAY...
f.close()             # ...and this line never runs. Leak + unflushed data.

If risky() raises, the exception propagates up and out of this block, sailing straight past f.close(). The descriptor leaks, and the buffered write may be lost. And this isn't an exotic edge case — anything can raise: a bad value, a full disk, a keyboard interrupt. A cleanup line you merely hope executes is not cleanup.

The context manager closes the hole structurally, so write it the way you've been told to:

with open("song.txt", "w") as f:
    f.write("My Favorite Song")
    risky()           # even if THIS raises...
# ...__exit__ still fires here — buffer flushed, fd slot freed — then the
# exception keeps propagating. Cleanup happened on the way out regardless.

Here is the actual mechanism, no magic. with open(...) as f binds the file object to f. More importantly, the with statement guarantees that f.__exit__() is called when the block ends — normally OR via an exception. The file object's __exit__ method calls f.close(), and that's the whole trick: __exit__ runs on every path out of the block. So a crash mid-write still flushes the buffer and still frees the descriptor before the exception continues on its way. with is treated as "law" not because teachers love tidiness, but because it moves cleanup from a line you hope executes to a guarantee the language enforces on every exit path.

You can prove __exit__ fires under an exception with your own eyes. Write a tiny class with __enter__ and __exit__ that prints when it exits. Wrap a deliberate crash in it, and watch EXIT called appear even as the traceback prints. Cleanup ran on the way out, and only then did the exception surface. Do the same with a real file and reopen it afterward. The bytes are there, flushed by the close that __exit__ guaranteed.

Traceback (most recent call last):
  File "save_playlist.py", line 3, in <module>
    with open("data/playlist.txt", "w", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'data/playlist.txt'

Traceback (most recent call last):
  File "loader.py", line 4, in <module>
OSError: [Errno 24] Too many open files: 'note.txt'

Traceback (most recent call last):
  File "save_note.py", line 4, in <module>
    f.write("Titanium\n")
ValueError: I/O operation on closed file.

# append_log.py -- 'a' never truncates. Run it twice and watch the file grow.
from datetime import datetime
from pathlib import Path

LOG = Path("plays.log")
PLAYED = ["Blinding Lights", "Titanium"]

stamp = datetime.now().strftime("%H:%M:%S")

with open(LOG, "a", encoding="utf-8") as f:      # 'a' -> create if absent, else seek to the end
    for title in PLAYED:
        f.write(f"{stamp}  played  {title}\n")

with open(LOG, "r", encoding="utf-8") as f:      # read the whole log back, every run
    lines = f.readlines()

print(f"--- {LOG.name} now holds {len(lines)} line(s) ---")
for line in lines:
    print(line.rstrip())

$ python append_log.py
--- plays.log now holds 2 line(s) ---
02:50:58  played  Blinding Lights
02:50:58  played  Titanium

$ python append_log.py          # a few seconds later, a NEW process
--- plays.log now holds 4 line(s) ---
02:50:58  played  Blinding Lights
02:50:58  played  Titanium
02:51:03  played  Blinding Lights
02:51:03  played  Titanium

All chapter you've been told the file is always bytes — yet open('song.txt') hands you a str, while open('song.txt','rb') hands you bytes. Same descriptor, same disk, same byte-only kernel. So where do the strings come from? →

# fix_the_forgotten_close.py -- the same job twice: once leaking, once guaranteed
from pathlib import Path

for i in range(1, 4):                                    # three little files to read
    Path(f"track{i}.txt").write_text(f"track {i}\n", encoding="utf-8")

PATHS = [f"track{i}.txt" for i in range(1, 4)]


# ---------- the program as found: opens, reads, never closes ----------
def load_leaky(paths):
    handles, rows = [], []
    for p in paths:
        f = open(p, "r", encoding="utf-8")               # no close(), no with
        handles.append(f)                                # the object stays alive...
        rows.append((f.fileno(), f.read().strip()))      # ...so the fd stays claimed
    return rows, handles


rows, handles = load_leaky(PATHS)
print("leaky :", rows)
print("        closed?", [h.closed for h in handles], "- three slots taken, none given back")
for h in handles:
    h.close()


# ---------- the repair: with, so __exit__ closes on every exit path ----------
def load_safe(paths):
    handles, rows = [], []
    for p in paths:
        with open(p, "r", encoding="utf-8") as f:
            handles.append(f)
            rows.append((f.fileno(), f.read().strip()))
    return rows, handles


rows, handles = load_safe(PATHS)
print("safe  :", rows)
print("        closed?", [h.closed for h in handles], "- one slot, reused, freed each time")


# ---------- and the reason it matters: an exception mid-loop ----------
def load_until_crash(paths, use_with):
    handles = []
    try:
        for p in paths:
            if use_with:
                with open(p, "r", encoding="utf-8") as f:
                    handles.append(f)
                    if p.endswith("2.txt"):
                        raise ValueError("bad row")
            else:
                f = open(p, "r", encoding="utf-8")
                handles.append(f)
                if p.endswith("2.txt"):
                    raise ValueError("bad row")          # jumps clean over the close below
                f.close()
    except ValueError:
        pass
    return [h.closed for h in handles]


print("crash, bare open():", load_until_crash(PATHS, False), "<- False = still open, leaked")
print("crash, with-block :", load_until_crash(PATHS, True), "<- closed anyway, on the way out")

# ------------- what it prints -------------
leaky : [(3, 'track 1'), (4, 'track 2'), (5, 'track 3')]
        closed? [False, False, False] - three slots taken, none given back
safe  : [(3, 'track 1'), (3, 'track 2'), (3, 'track 3')]
        closed? [True, True, True] - one slot, reused, freed each time
crash, bare open(): [True, False] <- False = still open, leaked
crash, with-block : [True, True] <- closed anyway, on the way out

06Text mode is a codec layer bolted onto a byte stream

Something in this chapter should have been quietly bothering you. Five sections of "the file is always bytes, the kernel only moves numbers." And yet the very first thing you ever did, open('song.txt'), handed you a str. Open the same file with 'rb' and you get bytes instead. But it is the same descriptor, the same disk, the same kernel that traffics only in numbers from 0 to 255. The bytes on the platter did not change. So where did the string come from? Something between the kernel and your variable is manufacturing str out of bytes. Until you can point at exactly which layer does it, the b'' prefix and every "works on my machine" encoding bug will feel like black magic.

Underneath every file mode there is one descriptor delivering raw bytes from the kernel. That is the ground truth, and it never changes. What changes is how many layers Python stacks on top of it. Open in binary mode — 'rb' or 'wb' — and Python gives you that byte stream almost bare: a BufferedReader or BufferedWriter sitting directly over the descriptor. read returns bytes, and write demands bytes. No interpretation happens, because none was asked for. This is the honest view of the file: exactly what the kernel moved.

Text mode — plain 'r' or 'w' — wraps that same buffered byte stream in one more object: an io.TextIOWrapper. And that wrapper is a codec layer, a two-way translator. On the way in, it decodes incoming bytes into a str using an encoding. On the way out, it encodes your str back into bytes before they ever reach the buffer and the write syscall. It also quietly translates newlines (\r\n on Windows ⇆ \n) so your text looks the same everywhere. The consequence is the sentence that dissolves the whole mystery: a str never touches the disk and never crosses the syscall boundary. Bytes do. The str exists only above the TextIOWrapper, conjured by the codec.

# count_file.py -- Word Count from chapter 2, now pointed at a real file
from collections import Counter
from pathlib import Path

SOURCE = Path("notes.txt")
SOURCE.write_text(
    "the jukebox plays a track and then it plays the next track\n"
    "every track is one line and every line is one track\n"
    "the file remembers what the process forgot\n",
    encoding="utf-8",
)

lines = words = chars = 0
tally = Counter()

with open(SOURCE, "r", encoding="utf-8") as f:
    for line in f:                                   # one line at a time: memory stays flat
        lines += 1
        chars += len(line)
        pieces = line.split()                        # split() on whitespace, ch 2
        words += len(pieces)
        tally.update(w.strip(".,!?'").lower() for w in pieces)

print(f"{SOURCE.name}: {lines} lines, {words} words, {chars} characters")
print("top 3      :", tally.most_common(3))

# the one-liner, for a file you already know is small
text = SOURCE.read_text(encoding="utf-8")
print("cross-check:", len(text.splitlines()), len(text.split()), len(text))

# ------------- what it prints -------------
notes.txt: 3 lines, 30 words, 154 characters
top 3      : [('the', 4), ('track', 4), ('plays', 2)]
cross-check: 3 30 154

See both branches at once, where the bytes on disk are identical and only the number of layers differs:

open("song.txt", "rb").read()   # b'My Favorite Song' — bytes, raw off the fd
open("song.txt", "r").read()    # 'My Favorite Song'  — str, decoded by the codec layer

The gap between them widens the moment a character needs more than one byte. Save a file containing a musical note ♪ or an emoji, and count. In 'rb' mode the length is larger than the number of visible characters. That one glyph is several bytes on disk (three, for ♪, in UTF-8). Read the same file in 'r' mode and the length equals the character count exactly, because the TextIOWrapper consumed those several bytes and produced one str element. That collapse — many bytes into one character — is the decode step. It happens at a layer you can now name and point to.

Which raises the sharpest, most practical point of the section: which encoding does text mode use? By default, whatever your platform's locale says. That's often UTF-8 on modern systems, but not always. And that "not always" is the source of a thousand "but it worked on my laptop" bugs. A file written as UTF-8 on your machine can be read with a different default encoding on a colleague's. It can decode into garbage or raise outright. The codec layer is powerful, but its default is a guess made by the platform. And guesses are where correctness goes to die.

you type
# encodings.py -- the argument you should never leave out
import locale
from pathlib import Path

print("this machine's text default:", locale.getpreferredencoding(False))

p = Path("title.txt")
with open(p, "w", encoding="utf-8") as f:            # say it. Every time.
    f.write("Café Bleu ♦".replace("é", "é"))

print("on disk      :", p.read_bytes())
print("len in bytes :", len(p.read_bytes()), "| len in characters:", len(p.read_text(encoding='utf-8')))

with open(p, "r", encoding="utf-8") as f:            # the right codec: the text comes back
    print("utf-8        :", repr(f.read()))

with open(p, "r", encoding="latin-1") as f:          # a WRONG codec that never complains
    print("latin-1      :", repr(f.read()), "<- no error. Just wrong.")

try:
    with open(p, "r", encoding="ascii") as f:        # a wrong codec that does complain
        f.read()
except UnicodeDecodeError as e:
    print("ascii        : UnicodeDecodeError at byte", e.start, "-", e.reason)

with open(p, "r", encoding="ascii", errors="replace") as f:
    print("ascii+replace:", repr(f.read()), "<- damage made visible, not silent")

with open(p, "rb") as f:                             # no codec at all: the kernel's own truth
    print("rb           :", f.read())

# the bug this argument prevents, caught in the act
try:
    with open("guessed.txt", "w") as f:              # no encoding= -> the platform guesses
        f.write("Café Bleu ♦".replace("é", "é"))
    print("no encoding= : wrote it, using", locale.getpreferredencoding(False))
    print("no encoding= : reads back as", repr(Path("guessed.txt").read_text(encoding="utf-8")))
except UnicodeEncodeError as e:
    print("no encoding= : UnicodeEncodeError -", e.reason, "- codec", e.encoding, "cannot spell", repr(e.object[e.start:e.end]))

you see
this machine's text default: cp1252
on disk      : b'Caf\xc3\xa9 Bleu \xe2\x99\xa6'
len in bytes : 14 | len in characters: 11
utf-8        : 'Café Bleu ♦'
latin-1      : 'Café Bleu â\x99¦' <- no error. Just wrong.
ascii        : UnicodeDecodeError at byte 3 - ordinal not in range(128)
ascii+replace: 'Caf�� Bleu ���' <- damage made visible, not silent
rb           : b'Caf\xc3\xa9 Bleu \xe2\x99\xa6'
no encoding= : UnicodeEncodeError - character maps to <undefined> - codec charmap cannot spell '♦'

And because the str only exists if the codec succeeds, you can watch the whole illusion refuse to form. Write bytes that are not valid UTF-8 in 'wb' mode. Then try to read that file in 'r' with encoding="utf-8", and you get a UnicodeDecodeError. Not corrupted text — no text at all, because the decoder could not build a legal str from those bytes. That error is the codec layer telling you, plainly, that the string was never real disk data. It was always something the top layer manufactured, and manufacturing can fail.

# backup.py -- copy a file to a .bak twin, byte for byte, in chunks
from pathlib import Path

src = Path("playlist.txt")
src.write_bytes(b"Blinding Lights|The Weeknd|200\nTitanium|David Guetta|245\n")

bak = src.with_suffix(src.suffix + ".bak")            # playlist.txt -> playlist.txt.bak
bak.unlink(missing_ok=True)

CHUNK = 64 * 1024                                     # one syscall per chunk, not per byte
copied = 0
with open(src, "rb") as fin, open(bak, "wb") as fout:  # 'b': no codec, nothing translated
    while True:
        block = fin.read(CHUNK)
        if not block:                                  # b"" means end of file
            break
        copied += fout.write(block)

print("source    :", src.name, "|", src.stat().st_size, "bytes")
print("backup    :", bak.name, "|", bak.stat().st_size, "bytes |", copied, "copied")
print("identical :", src.read_bytes() == bak.read_bytes())

print("--- backup contents, read back ---")
with open(bak, "r", encoding="utf-8") as f:            # a second read proves it, we don't claim it
    print(f.read(), end="")
print("----------------------------------")

# never silently clobber a backup you might need: 'x' refuses
try:
    with open(bak, "x", encoding="utf-8") as f:
        f.write("clobbered")
except FileExistsError:
    print("second run:", bak.name, "already exists - 'x' refused to overwrite it")

# ------------- what it prints -------------
source    : playlist.txt | 57 bytes
backup    : playlist.txt.bak | 57 bytes | 57 copied
identical : True
--- backup contents, read back ---
Blinding Lights|The Weeknd|200
Titanium|David Guetta|245
----------------------------------
second run: playlist.txt.bak already exists - 'x' refused to overwrite it

You've crossed the first boundary: bytes leaving your process, over the ring wall, onto remembering hardware, and back again — with a codec layer deciding whether they come home as bytes or as text. That codec is a whole world of its own. Chapter 17 climbs inside it: what an encoding actually is, why UTF-8 won, and how a handful of bytes agree to mean a single character. →

# jukebox_store.py -- chapter 14's playlist, now surviving the process that made it
from dataclasses import dataclass
from pathlib import Path

DB = Path("jukebox.txt")
SEP = "|"


@dataclass
class Track:                                     # ch 12/14, unchanged
    title: str
    artist: str
    seconds: int
    plays: int = 0

    def __str__(self):
        minutes, secs = divmod(self.seconds, 60)
        return f"{self.title} - {self.artist} ({minutes}:{secs:02d})  played {self.plays}x"


def save(tracks, path=DB):
    """One track per line, fields joined by |. Returns how many lines landed."""
    with open(path, "w", encoding="utf-8") as f:          # 'w': this file IS the playlist now
        for t in tracks:
            f.write(SEP.join([t.title, t.artist, str(t.seconds), str(t.plays)]) + "\n")
    return len(tracks)


def load(path=DB):
    """Text back into Track objects. Bad lines are reported, never guessed at."""
    tracks = []
    if not path.exists():                                  # first ever run: nothing to load
        return tracks
    with open(path, "r", encoding="utf-8") as f:
        for lineno, raw in enumerate(f, start=1):
            line = raw.rstrip("\n")
            if not line:
                continue
            try:
                title, artist, seconds, plays = line.split(SEP)
                tracks.append(Track(title, artist, int(seconds), int(plays)))
            except ValueError as e:
                print(f"  line {lineno} skipped: {line!r} -> {e}")
    return tracks


# ---------- run 1: build it in RAM, play one, save it ----------
playlist = [
    Track("Blinding Lights", "The Weeknd", 200),
    Track("Titanium", "David Guetta", 245),
    Track("Levels", "Avicii", 203),
]
playlist[0].plays += 1                                     # somebody pressed play
print("in memory :", len(playlist), "tracks")
for t in playlist:
    print("   ", t)
print("saved     :", save(playlist), "lines ->", DB.name)

print("--- what is actually on disk ---")
with open(DB, "r", encoding="utf-8") as f:                 # read it back to SHOW it, not claim it
    print(f.read(), end="")
print("-------------------------------")

# ---------- the process ends here. Everything in RAM is gone. ----------
del playlist

# ---------- run 2: a fresh program, loading from the file ----------
restored = load()
print("loaded    :", len(restored), "tracks, rebuilt as", type(restored[0]).__name__, "objects")
for t in restored:
    print("   ", t)
print("plays kept:", restored[0].plays, "- the play we recorded survived the process")
print("round trip:", restored == [
    Track("Blinding Lights", "The Weeknd", 200, 1),
    Track("Titanium", "David Guetta", 245),
    Track("Levels", "Avicii", 203),
])

# ---------- the honest limit of a delimiter you chose yourself ----------
with open(DB, "a", encoding="utf-8") as f:
    f.write("Song|With|A|Pipe|In It|0\n")
print("re-loaded :", len(load()), "tracks")

in memory : 3 tracks
    Blinding Lights - The Weeknd (3:20)  played 1x
    Titanium - David Guetta (4:05)  played 0x
    Levels - Avicii (3:23)  played 0x
saved     : 3 lines -> jukebox.txt
--- what is actually on disk ---
Blinding Lights|The Weeknd|200|1
Titanium|David Guetta|245|0
Levels|Avicii|203|0
-------------------------------
loaded    : 3 tracks, rebuilt as Track objects
    Blinding Lights - The Weeknd (3:20)  played 1x
    Titanium - David Guetta (4:05)  played 0x
    Levels - Avicii (3:23)  played 0x
plays kept: 1 - the play we recorded survived the process
round trip: True
  line 4 skipped: 'Song|With|A|Pipe|In It|0' -> too many values to unpack (expected 4)
re-loaded : 3 tracks