20Inside import — sys.path, finders, loaders, and .pyc

In Chapter 19 we met import from the outside — four honest verbs, find, compile, exec, cache, named but still half-trusted. In this one we pry the lid off and watch each one run. Here's the plan. We print sys.path, the ordered list of folders import walks. We watch a stray random.py in your own directory beat the entire standard library. Then we follow the winning file through the finder/loader handshake, into the .pyc bytecode cache with its sixteen-byte header, and out the far side as a live module object. The whole way through, we keep asking the one thing that matters — when you type import X, what exactly decides which bytes on disk become the module you get back? By the end, import is no longer a keyword you trust but a machine you can single-step. It is an ordered search you can print, a finder/loader protocol you could implement yourself, and a compile pipeline that turns the text you type into the same bytecode the Volume 1 VM already knows how to run. It is the same story as every boundary in this volume. What crosses from disk into your process is always bytes, and import is the machine that rebuilds live objects on the near side.

01sys.path: an ordered search, first match wins

Let's start with the bug everyone hits. You're debugging something fiddly with randomness, so you make a throwaway file to poke at it and name it, reasonably, random.py. Run your real program and watch it detonate: AttributeError: module 'random' has no attribute 'random'. You did not touch the standard library. You did not mistype. And yet import random has handed you your file instead of Python's. The very same code that works on your laptop will explode the moment it lands in a directory that happens not to contain your little file. Every Python programmer meets this bug, stares at it for an hour, and walks away convinced import is haunted. It is not. Import is a deterministic walk down a list you can print, and the instant you can see the list, the haunting stops for good.

Here is the whole secret in one sentence. import X does not search your disk and it does not know where the standard library lives. It walks sys.path, a plain list[str] of directory names, from top to bottom. It asks each location the same blunt question, "do you have X?", and it stops at the first one that says yes. First match wins. That rule is the entire mystery. So print the list and look at the thing that has been deciding your imports all along:

>>> import sys
>>> for i, p in enumerate(sys.path):
...     print(i, repr(p))
0 ''                                  # cwd / the script's own folder — checked FIRST
1 'C:\\Python312\\python312.zip'
2 'C:\\Python312\\Lib'                # the standard library lives here
3 'C:\\Python312'
4 'C:\\Python312\\Lib\\site-packages'  # where pip installs land

Read the entries in the order the interpreter built them at startup, because the order is the rule. Index 0 is the special one. Run python foo.py and it becomes the folder that foo.py lives in. Run -c, or -m, or the bare REPL and it becomes '', meaning the current working directory. That single entry, checked before everything else, is exactly why a sibling random.py beats the standard library — your project folder is first in line. After it come the $PYTHONPATH entries you set in the environment. Then come the standard-library directories and the frozen modules baked into the executable. And finally the site-packages directory that the site module appends, where every pip install deposits its wares. Top to bottom, first match wins, no exceptions.

you type
# ---------- wheres_my_import.py ----------
# wheres_my_import.py -- print the ordered list that decides every import
import os
import sys
import sysconfig

STDLIB = sysconfig.get_paths()["stdlib"]


def label(i, entry):
    if i == 0:
        return "the launched script's folder (or the cwd)"
    if entry.endswith(".zip"):
        return "the standard library, zipped"
    if entry == STDLIB:
        return "the standard library"
    if entry.endswith("site-packages"):
        return "site-packages -- every pip install lands here"
    if entry == sys.prefix:
        return "this environment's own folder"
    if entry == sys.base_prefix or entry.endswith("DLLs"):
        return "the interpreter this one was built from"
    return "an extra entry (PYTHONPATH, or a .pth file)"


print("python", sys.version.split()[0], "| entries:", len(sys.path))
for i, entry in enumerate(sys.path):
    print(f"{i}  {os.path.basename(entry) or '(cwd)':<14} {label(i, entry)}")


# ---------- the terminal ----------
$ python wheres_my_import.py
$ python -c "import beat"
$ set PYTHONPATH=C:\tmp\pathbox\tools
$ python -c "import sys, beat; print('BPM', beat.BPM); print('sys.path[1]', sys.path[1])"

you see
$ python wheres_my_import.py
python 3.12.7 | entries: 7
0  shop           the launched script's folder (or the cwd)
1  python312.zip  the standard library, zipped
2  DLLs           the interpreter this one was built from
3  Lib            the standard library
4  anaconda3      the interpreter this one was built from
5  .venv          this environment's own folder
6  site-packages  site-packages -- every pip install lands here

$ python -c "import beat"
ModuleNotFoundError: No module named 'beat'

$ set PYTHONPATH=C:\tmp\pathbox\tools
$ python -c "import sys, beat; ..."
BPM 128
sys.path[1] C:\tmp\pathbox\tools

you type
# ---------- your folder ----------
jukebox/
    random.py          # the five-minute experiment you meant to delete
    shuffle_demo.py    # the program that suddenly broke

# ---------- random.py (yours) ----------
# random.py -- a five-minute experiment you meant to delete
COIN = ["heads", "tails"]

# ---------- shuffle_demo.py ----------
# shuffle_demo.py -- pick a track at random
import random

TRACKS = ["Blinding Lights", "Titanium", "Levels"]
random.seed(20)
print(random.choice(TRACKS))


# ---------- the terminal ----------
$ python shuffle_demo.py
$ python -c "import random; print(random.__file__)"
$ mv random.py coinflip.py              # ren random.py coinflip.py on Windows cmd
$ python shuffle_demo.py

you see
$ python shuffle_demo.py
Traceback (most recent call last):
  File "C:\tmp\jukebox\shuffle_demo.py", line 5, in <module>
    random.seed(20)
    ^^^^^^^^^^^
AttributeError: module 'random' has no attribute 'seed'

$ python -c "import random; print(random.__file__)"
C:\tmp\jukebox\random.py

$ mv random.py coinflip.py
$ python shuffle_demo.py
Levels

The most liberating fact is that sys.path is just data. It is a list. You can sys.path.insert(0, ...), .append(...), or reorder it at runtime, and the very next import obeys the new order. Import re-reads the list on every call. It never snapshots it. But before the walk even begins, import checks a faster cache: sys.modules, the dictionary of everything already imported in this process. A hit there means no search at all, and the module object is returned instantly. That is why the second import json is nearly free, and why a module's top-level code runs exactly once per process no matter how many files import it.

you type
# ---------- beat.py ----------
# beat.py -- a module that announces itself when it executes
print("beat.py: top level running")

BPM = 128


# ---------- cache_probe.py ----------
# cache_probe.py -- the import cache, used as a debugging instrument
import os
import sys

print("loaded before we start :", len(sys.modules), "modules")
print("beat in the cache?     :", "beat" in sys.modules)

import beat                        # MISS -- find, compile, exec; the print fires
import beat                        # HIT  -- one dict lookup, and silence

print("beat in the cache?     :", "beat" in sys.modules)
print("cache holds the object :", sys.modules["beat"] is beat)
print("which file won         :", os.path.basename(sys.modules["beat"].__file__))
print("BPM                    :", beat.BPM)

beat.BPM = 200                     # you patch the live module
del sys.modules["beat"]            # evict it: the next import must do real work
import beat                        # MISS again -- the print fires a second time

print("BPM after re-import    :", beat.BPM)
print("sys itself has no file :", hasattr(sys, "__file__"))

you see
$ python cache_probe.py
loaded before we start : 35 modules
beat in the cache?     : False
beat.py: top level running
beat in the cache?     : True
cache holds the object : True
which file won         : beat.py
BPM                    : 128
beat.py: top level running
BPM after re-import    : 128
sys itself has no file : False

>>> import random
>>> random.__file__
'C:\\Users\\you\\project\\random.py'   # not the stdlib — YOUR file won the search
>>> del sys.modules['random']        # evict the wrong module from the cache
>>> # now delete your local random.py, then import again → it falls through to the stdlib

The walk finds a directory that can produce the name — but who decides whether a folder "has" the module, and who turns the winning file into a live object? Those are two different jobs, done by two different kinds of object, and pulling them apart is what turns import from a keyword into a machine. Finders and loaders →

02Finders and loaders: the import protocol

You have watched import pull a module out of a .zip file. You have seen a Jupyter notebook import a sibling notebook that has no .py at all. If import were simply "open a .py and run it," none of that could exist. So there must be a seam — a place where "how do we find this?" is cleanly separated from "how do we build it?". Only a separated, swappable design could serve modules from a zip, a notebook, a database row, or thin air. Finding that seam is what turns import from a magic word into a machine with named parts you could implement yourself.

The real shape of import X is a two-phase protocol driven by importlib. Phase FIND: the machinery iterates sys.meta_path, a list of meta path finders, and asks each one find_spec(name, path, target). A finder returns either None, meaning "not mine, ask the next one," or a ModuleSpec. The three defaults, in order, are BuiltinImporter (the C builtins from the last section), FrozenImporter (modules frozen into the binary), and PathFinder, the one that finally consults sys.path. Print the chain and it is exactly what the mechanism predicts:

>>> import sys, json
>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>,
 <class '_frozen_importlib.FrozenImporter'>,
 <class '_frozen_importlib_external.PathFinder'>]
>>> json.__spec__
ModuleSpec(name='json', loader=<...SourceFileLoader...>,
           origin='C:\\Python312\\Lib\\json\\__init__.py',
           submodule_search_locations=['C:\\Python312\\Lib\\json'])

That ModuleSpec is the pivot of the whole system — the blueprint the rest of import consumes. Learn its fields, because they answer every "how did import know…" question you will ever have. .name is the dotted name. .loader is the object that will actually build the module. .origin is usually the source filename (or the literal string '(built-in)'). .submodule_search_locations is the __path__ a package hands to its children. .cached points at the .pyc. A finder's entire job is to fill in that form and hand it back.

you type
# ---------- your folder ----------
check/
    csv.py             # a quick parser you wrote for one CSV, months ago


# ---------- the terminal ----------
$ python -c "import analytics"
$ python -c "import importlib.util as u; print(u.find_spec('analytics'))"
$ python -c "import importlib.util as u; print(u.find_spec('csv').origin)"
$ python -c "import sys; print('csv' in sys.stdlib_module_names)"
$ python -c "import sys; print(repr(sys.path[0]))"

you see
$ python -c "import analytics"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'analytics'

$ ... print(u.find_spec('analytics'))
None

$ ... print(u.find_spec('csv').origin)
C:\tmp\check\csv.py

$ ... print('csv' in sys.stdlib_module_names)
True

$ ... print(repr(sys.path[0]))
''

Phase LOAD is where the spec becomes a living thing. Import calls importlib.util.module_from_spec(spec) to make an empty module object. It inserts that object into sys.modules before running a single line of it. This is the subtle move that lets circular imports partially work: a module that imports back into a half-built one finds the half-built object already registered, rather than triggering an infinite re-import. Only then does import call spec.loader.exec_module(module), which runs the module's code object into the module's __dict__. The division of labour is crisp. A finder answers "where and how," and a loader does "create and exec." Nothing is hidden. You can perform the entire dance by hand:

import importlib.util, sys

def manual_import(name):
    spec = importlib.util.find_spec(name)          # PHASE FIND → a ModuleSpec
    module = importlib.util.module_from_spec(spec)  # create the empty module
    sys.modules[name] = module                      # register BEFORE executing
    spec.loader.exec_module(module)                 # PHASE LOAD → run the body
    return module

textwrap = manual_import('textwrap')
print(textwrap.shorten("the whole import protocol, by hand", 20))

One layer remains, and it explains the zip trick. PathFinder does not read files itself. For each entry on sys.path it consults sys.path_hooks to obtain a path entry finder for that specific location. The default hook handles ordinary directories, and another handles zip archives. It memoises the result in sys.path_importer_cache, so it only computes each one once per directory. So there are genuinely two finder layers: meta path finders are global strategies, and path entry finders are per-sys.path-entry workers. That is why a .zip on sys.path "just works": a path hook claims the archive and returns a finder that reads modules out of it.

# ---------- the broken layout ----------
jukebox/
    app.py                 # imports json and playlist
    json.py                # a helper you wrote last month  -- SHADOW
    lib/
        playlist.py        # the module app.py wants        -- WRONG FOLDER
    import_detective.py

# ---------- app.py ----------
# app.py -- print the playlist as JSON
import json
import playlist

print(json.dumps(playlist.load()))

# ---------- lib/playlist.py ----------
# playlist.py -- the data the app prints
TRACKS = [
    {"title": "Blinding Lights", "seconds": 200},
    {"title": "Titanium", "seconds": 245},
]


def load():
    return TRACKS


# ---------- import_detective.py ----------
# import_detective.py -- answer "why can't Python import that?" in four checks
import importlib.util
import os
import sys

HERE = os.path.realpath(os.getcwd())


def short(path):
    """A readable path: relative to the project when the file is inside it."""
    real = os.path.realpath(path)
    if real.startswith(HERE):
        return os.path.relpath(real, HERE).replace(os.sep, "/")
    return "/".join(real.replace(os.sep, "/").split("/")[-2:])


def files_named(name):
    """Every <name>.py under the project -- on sys.path or not."""
    hits = []
    for root, dirs, files in os.walk(HERE):
        dirs[:] = [d for d in dirs if d not in {"__pycache__", ".git", ".venv"}]
        if name + ".py" in files:
            hits.append(short(os.path.join(root, name + ".py")))
    return hits


def investigate(name):
    print("---", name, "---")
    print("1 in sys.modules :", name in sys.modules)

    spec = importlib.util.find_spec(name)
    if spec is None:
        print("2 find_spec      : None -- no sys.path entry answers to that name")
        hits = files_named(name)
        if hits:
            print("3 file on disk   :", ", ".join(hits))
            print("  VERDICT       : right file, wrong folder -- it is not on sys.path")
        else:
            print("3 file on disk   : none -- check the spelling, or pip install it")
            print("  VERDICT       : the name does not exist here")
        return

    origin = spec.origin or "(built-in)"
    mine = os.path.realpath(origin).startswith(HERE)
    print("2 find_spec      :", short(origin))
    print("3 whose file     :", "yours" if mine else "not yours")
    if mine and name in sys.stdlib_module_names:
        print("  VERDICT       : SHADOW -- your file beats the stdlib module of that name")
    else:
        print("  VERDICT       : import will succeed with that file")


for wanted in sys.argv[1:]:
    investigate(wanted)

$ python app.py
Traceback (most recent call last):
  File "C:\tmp\jukebox\app.py", line 3, in <module>
    import playlist
ModuleNotFoundError: No module named 'playlist'

$ python import_detective.py playlist json
--- playlist ---
1 in sys.modules : False
2 find_spec      : None -- no sys.path entry answers to that name
3 file on disk   : lib/playlist.py
  VERDICT       : right file, wrong folder -- it is not on sys.path
--- json ---
1 in sys.modules : False
2 find_spec      : json.py
3 whose file     : yours
  VERDICT       : SHADOW -- your file beats the stdlib module of that name

$ mv lib/playlist.py playlist.py     &&  rmdir lib
$ mv json.py pairs.py

$ python app.py
[{"title": "Blinding Lights", "seconds": 200}, {"title": "Titanium", "seconds": 245}]

$ python import_detective.py playlist json
--- playlist ---
1 in sys.modules : False
2 find_spec      : playlist.py
3 whose file     : yours
  VERDICT       : import will succeed with that file
--- json ---
1 in sys.modules : False
2 find_spec      : json/__init__.py
3 whose file     : not yours
  VERDICT       : import will succeed with that file

The loader ran the module's code object — but where did that code object come from, and why is the second import so much faster than the first? The answer is a folder you have seen appear next to your code and always been slightly afraid to delete. __pycache__ →

03Source becomes bytecode; __pycache__ caches it

The first time you import a large library there is a perceptible pause. The second time it is instant. And sitting next to your code, unbidden, is a folder called __pycache__ full of .pyc files. You have wondered three things about it and never quite asked out loud: is it safe to delete, why does it exist, and did my source just get "compiled" like C? Answer those and you have demystified startup time, an entire family of "stale bytecode" bugs, and exactly what Python is trading memory for speed on.

you type
# ---------- your folder ----------
# beat.py    -- a library module, IMPORTED by player.py
# player.py  -- the program you launch

# ---------- player.py ----------
# player.py -- the program you launch
import beat

print("BPM is", beat.BPM)


# ---------- the terminal ----------
$ ls
$ python player.py
$ ls __pycache__
$ rm -rf __pycache__
$ python -B player.py
$ ls

you see
$ ls
beat.py
player.py

$ python player.py
beat.py: top level running
BPM is 128

$ ls __pycache__
beat.cpython-312.pyc        # beat was IMPORTED. player.py was RUN -- no .pyc.

$ rm -rf __pycache__
$ python -B player.py
beat.py: top level running
BPM is 128

$ ls
beat.py
player.py                   # -B wrote nothing

Python is compiled — just not to machine code. Every time a .py is imported, its source is compiled into a code object full of bytecode. (The one exception is the script you run directly as __main__; that one is never cached.) To avoid paying that compile cost on every future import, the loader marshals the code object to bytes and writes them to __pycache__/NAME.cpython-3XX.pyc. The whole trick lives in that file's header, so let us take it apart byte by byte. A modern .pyc begins with sixteen bytes. First comes a 4-byte magic number that encodes the exact CPython version and bytecode format. This is why the filename is tagged cpython-312, and why a 3.11 .pyc is silently ignored by 3.12. Next comes a 4-byte bit-field of flags. Then, in the default timestamp scheme, come the source's 4-byte modification time plus its 4-byte size.

Now the payoff. On the next import the loader reads only that header. It compares the magic number: mismatch means a different interpreter, so recompile. If the magic matches, in timestamp mode it compares the stored mtime and size against a fresh os.stat of the source. If they match, it concludes "the source is unchanged, trust the cache," skips tokenizing, parsing and compiling entirely, and simply marshal.loads the code object straight back into memory to execute. That is why the second import is fast: the expensive front of the pipeline is skipped, and only unmarshal-plus-exec remain. Decode the header yourself and the machine stops being mysterious:

import struct, importlib.util, os

with open("__pycache__/mymod.cpython-312.pyc", "rb") as f:
    magic = f.read(4)
    flags = struct.unpack("<I", f.read(4))[0]
    mtime = struct.unpack("<I", f.read(4))[0]
    size  = struct.unpack("<I", f.read(4))[0]

print(magic == importlib.util.MAGIC_NUMBER)      # True — this interpreter's format
print(mtime, int(os.stat("mymod.py").st_mtime))  # equal → cache is trusted

Be exact about the failure mode, because it bites real builds. Timestamp invalidation trusts the filesystem clock. Restore an older source whose mtime is now newer than the cache, or build on a machine with a skewed clock, and Python can serve you stale bytecode — running code that no longer matches the source you are reading. That is precisely why hash-based .pyc files exist (PEP 552). When the flags bit is set, those mtime and size fields are replaced by a truncated SipHash of the source, and the loader validates by content rather than by clock. You opt in with py_compile using CHECKED_HASH (rehash and compare every time) or UNCHECKED_HASH (trust the hash blindly, for reproducible builds where the toolchain guarantees freshness).

# pycache_watch.py -- watch the .pyc be trusted, go stale, and be rebuilt
import importlib.util
import os
import shutil
import struct
import subprocess
import sys
from pathlib import Path

SRC = Path("beat.py")
PYC = Path(importlib.util.cache_from_source(str(SRC)))   # the exact cache path

SRC.write_text('print("beat.py: top level running")\n\nBPM = 128\n', encoding="utf-8")


def header_mtime():
    """Bytes 8-12 of the .pyc: the source mtime it claims it was built from."""
    magic, flags, mtime, size = struct.unpack("<4sIII", PYC.read_bytes()[:16])
    return mtime


def fresh_import():
    """A brand-new process, so sys.modules starts empty and import really runs."""
    done = subprocess.run([sys.executable, "-c", "import beat; print('BPM =', beat.BPM)"],
                          capture_output=True, text=True)
    for line in done.stdout.splitlines():
        print("        |", line)


shutil.rmtree(PYC.parent, ignore_errors=True)
print("cold    : .pyc exists?", PYC.exists())

print("run 1   : the first import ever")
fresh_import()
m1 = header_mtime()
print("          wrote", PYC.name)
print("          header mtime == source mtime :", m1 == int(SRC.stat().st_mtime))

print("run 2   : same source, brand-new process")
fresh_import()
m2 = header_mtime()
print("          header mtime changed?        :", m2 != m1, "-- COMPILE was skipped")

SRC.write_text(SRC.read_text(encoding="utf-8").replace("128", "174"), encoding="utf-8")
os.utime(SRC, (SRC.stat().st_atime + 2, SRC.stat().st_mtime + 2))
print("edit    : BPM 128 -> 174, saved")
print("          header mtime == source mtime :", m2 == int(SRC.stat().st_mtime), "-- STALE")

print("run 3   : after the edit")
fresh_import()
m3 = header_mtime()
print("          header mtime changed?        :", m3 != m2, "-- it recompiled")
print("          header mtime == source mtime :", m3 == int(SRC.stat().st_mtime))

$ python pycache_watch.py
cold    : .pyc exists? False
run 1   : the first import ever
        | beat.py: top level running
        | BPM = 128
          wrote beat.cpython-312.pyc
          header mtime == source mtime : True
run 2   : same source, brand-new process
        | beat.py: top level running
        | BPM = 128
          header mtime changed?        : False -- COMPILE was skipped
edit    : BPM 128 -> 174, saved
          header mtime == source mtime : False -- STALE
run 3   : after the edit
        | beat.py: top level running
        | BPM = 174
          header mtime changed?        : True -- it recompiled
          header mtime == source mtime : True

The cache stores a code object — the compiled unit the loader executes. But where does that object come from? Between the characters you type and the bytecode the VM runs sit three transformations, and the next section freeze-frames every one of them. Text → AST → code object →

04The compile pipeline: source → AST → code object

Volume 1 handed you a virtual machine that eats bytecode and pushes values on a stack. But you do not write bytecode — you write text. Somewhere between the characters your fingers produce and the instructions the VM consumes, three transformations happen. While they stay invisible, the VM feels disconnected from the language you actually speak. This section closes the loop. It shows the exact assembly line that turns total = a + b into the LOAD_FAST/BINARY_OP/STORE_FAST instructions your Volume 1 stack machine already knows how to run. And it hands you two tools, ast and dis, to stop the line at any station and stare.

# save BOTH files to disk first, then run the second one

# ---------- halfgood.py ----------
print('the first line ran')
BPM = 128
print('and the third'

# ---------- main.py ----------
print("nothing has complained yet")   # the file with a hole in it is already on disk
import halfgood                       # ...and only NOW does it reach the compiler

$ python main.py

nothing has complained yet
Traceback (most recent call last):
  File "C:\tmp\jukebox\main.py", line 2, in <module>
    import halfgood
  File "C:\tmp\jukebox\halfgood.py", line 3
    print('and the third'
         ^
SyntaxError: '(' was never closed

Station 1 — TOKENIZE. The tokenizer sweeps the raw source string into a stream of typed tokens: NAME, OP, NUMBER, NEWLINE, and — the detail that explains a whole category of beginner pain — INDENT and DEDENT. Python's significant indentation is not a special case buried in the parser. It is literal tokens the lexer synthesises when the leading whitespace changes. That is why mixing tabs and spaces is a tokenizer-level error, thrown before the grammar is ever consulted. Watch the stream directly:

import tokenize, io
src = "total = a + b\n"
for tok in tokenize.generate_tokens(io.StringIO(src).readline):
    print(tokenize.tok_name[tok.type], repr(tok.string))
# NAME 'total' · OP '=' · NAME 'a' · OP '+' · NAME 'b' · NEWLINE '\n' · ENDMARKER ''

Station 2 — PARSE to an AST. Since 3.9 CPython uses a PEG parser that assembles those tokens into an Abstract Syntax Tree. That is a tree of typed nodes where precedence is baked into the shape and all formatting is gone. total = a + b becomes an Assign whose target is Name('total') and whose value is a BinOp(Name('a'), Add(), Name('b')). This is the same tree that decorators, linters, and this book's own tracer inspect — structure without a single stray space or comment:

import ast
print(ast.dump(ast.parse("total = a + b"), indent=2))
# Module(body=[
#   Assign(targets=[Name(id='total', ctx=Store())],
#          value=BinOp(left=Name(id='a'), op=Add(), right=Name(id='b')))])

Station 3 — COMPILE to a code object. The compiler walks the AST, builds a control-flow graph, and emits bytecode into a code object (types.CodeType). This is the real unit of executable Python — the thing a function has (func.__code__) and a module is. Its fields are the accumulated payload of everything upstream: co_code (the raw bytecode bytes), co_consts (literals and nested code objects), co_names (global and attribute names), co_varnames (locals), co_argcount, co_flags, and a line-number table mapping bytecode offsets back to source lines for tracebacks. Then dis.dis disassembles co_code into readable opcodes. And here the whole book snaps together, because these are the exact stack instructions from Volume 1:

import dis
def f(a, b):
    total = a + b
    return total
dis.dis(f)
# LOAD_FAST    a       push local a onto the value stack        stack: [a]
# LOAD_FAST    b       push local b                             stack: [a, b]
# BINARY_OP    +       pop two, push their sum                  stack: [445]
# STORE_FAST   total   pop into the local `total`               stack: []
# LOAD_FAST    total   push it again to return                  stack: [445]
# RETURN_VALUE         pop and hand it back to the caller

And now the tie that binds the chapter: compile(src, name, 'exec') is literally stations 1 through 3 in a single call, returning the code object. Import's compile step is this pipeline. The .pyc from the last section caches its station-3 output, and the loader from section two runs that output into a module's __dict__. One unbroken arc: text → tokens → AST → code object → (marshalled into a .pyc) → executed by the VM. compile even hands you a code object you can eval directly:

code = compile("a + b", "<demo>", "eval")   # tokenize + parse + compile
print(code.co_names)                       # ('a', 'b')
print(eval(code, {"a": 200, "b": 245}))    # 445 — Titanium meets Blinding Lights again

You now know how one file becomes a module. But a real codebase splits one logical package across many folders — and the rule you were taught, "a package needs an __init__.py," turns out to be only half true. Namespace packages →

05Namespace packages: folders without __init__.py

Someone taught you a firm rule: a package needs an __init__.py. Then you clone a large monorepo where import company.billing and import company.auth resolve to two completely different directories. And there is no __init__.py in sight, yet every import succeeds. Either the rule you learned is wrong, or something deeper is running underneath it. Something deeper is running underneath it, and it is exactly how big organisations split one logical package across many folders, repositories, and separately installed distributions.

There are two kinds of package, and the difference is one file. A regular package is a directory containing __init__.py. Importing it runs that file and sets the package's __path__ to a single-element list — the one directory it came from. A namespace package (PEP 420, since 3.3) is what you get when the import system finds directories matching the name across sys.path but none of them contains __init__.py. Here is the precise algorithm PathFinder runs while scanning for company. If an entry has company/__init__.py, that is a regular package — stop. If an entry has a plain directory company/ with no __init__.py, record that directory as a portion and keep scanning the rest of sys.path, collecting every matching directory. If the scan ends without ever finding a regular package or a company.py module, import synthesises a namespace package. Its __path__ is not a plain list but a live _NamespacePath object holding all the collected portions.

dirA/company/billing.py      # no __init__.py anywhere under company/
dirB/company/auth.py         # a second, separate portion of the same name

>>> import sys
>>> sys.path[:0] = ['dirA', 'dirB']
>>> import company.billing, company.auth   # both succeed
>>> company.__file__                        # None — no single file backs it
>>> type(company.__path__)
<class '_frozen_importlib_external._NamespacePath'>
>>> list(company.__path__)
['dirA/company', 'dirB/company']                 # both portions, merged

That merged __path__ is the entire mechanism. import company from dirA and import company.auth where auth lives in dirB both work, because company.__path__ spans both directories, and submodule search walks every portion. Contrast the two sharply. A regular package is one directory that runs __init__.py — a real place to put initialisation code, an __all__, side effects, a version string. A namespace package is zero-or-many directories, runs no code, has __file__ set to None, and carries a live multi-location __path__ that is recomputed lazily as sys.path changes.

The design tradeoff is worth naming, because it is why the feature exists at all. Namespace packages let separately installed distributions contribute subpackages under one shared top-level name. The classic case is pip install azure-storage and pip install azure-identity landing in different site-packages folders, yet both importable as azure.storage and azure.identity. You gain that composability. You lose a place to put package-init code. And you inherit a subtler hazard. An accidental empty folder named like a package cannot beat a real installed one — a regular package with __init__.py still wins the scan. What the stray folder does is worse in a quieter way: when the real package is missing (a typo'd install, a broken checkout), the import that should fail loudly now "succeeds" as an empty namespace package. The tell is always the same pair of fields:

>>> import json                 # a regular package
>>> json.__file__                # '.../Lib/json/__init__.py' — a real file
>>> json.__path__                # ['.../Lib/json'] — a plain list, one entry
>>> # namespace package: __file__ is None, type(__path__) is _NamespacePath

You have now seen finders on a list, a spec protocol, and a default filesystem implementation with two flavours of package. The obvious, powerful question remains: if finders are just objects on a list — can you add one? Import hooks →

06Import hooks: why the protocol is pluggable

You know the shape now: import is find_spec → module_from_spec → exec_module, and finders live on a plain Python list. Which raises the question this whole chapter has been walking toward — if it is just objects on a list, can I add my own? Yes. And that yes is the entire difference between a language feature you use and a mechanism you own. Import hooks are how tools import notebooks as modules, how packages get lazily loaded, how encrypted or remote code gets fetched and run. And, more to the point, they are how you would build any of those yourself, in pure Python, at runtime.

There are two extension points, and choosing between them is choosing your blast radius. Meta path hooks are the sledgehammer. Put a finder object on sys.meta_path, and because that list is consulted first and for every single import, your finder can intercept any name from any source — a database row, a URL, an in-memory dictionary, a decrypted blob. It needs one method, find_spec(name, path, target), returning a ModuleSpec or None to politely decline. Here is the minimal shape: a finder that serves modules whose source is just a string in memory, with no file behind them at all:

import sys, importlib.util, importlib.abc

SOURCES = {"greet": "def hello():\n    return 'hi from memory'\n"}

class DictLoader(importlib.abc.Loader):
    def __init__(self, src): self.src = src
    def create_module(self, spec): return None          # use the default empty module
    def exec_module(self, module):
        code = compile(self.src, module.__name__, "exec")  # the section-4 pipeline!
        exec(code, module.__dict__)                     # run into the module namespace

class DictFinder(importlib.abc.MetaPathFinder):
    def find_spec(self, name, path, target=None):
        if name in SOURCES:
            return importlib.util.spec_from_loader(name, DictLoader(SOURCES[name]))
        return None                                    # decline → the next finder is asked

sys.meta_path.insert(0, DictFinder())
import greet                                        # no file on disk exists
print(greet.hello())                                 # hi from memory

Notice what you just did. Import fetched source from a Python dict, ran it through compile — the exact pipeline from section four — and executed the resulting code object into a module's __dict__, the exact load phase from section two. Every piece of this chapter reappears, now under your control.

# ---------- report/json.py  (YOURS -- the shadow) ----------
# json.py -- a helper you wrote last week and forgot about
def as_rows(pairs):
    return [{"name": k, "n": v} for k, v in pairs]


# ---------- report/save_report.py ----------
# save_report.py -- write the play counts to disk
import json

ROWS = [{"track": "Titanium", "plays": 12}]

with open("report.json", "w", encoding="utf-8") as f:
    json.dump(ROWS, f)

print("wrote report.json")


# ------------- $ python save_report.py -------------
Traceback (most recent call last):
  File "C:\tmp\report\save_report.py", line 7, in <module>
    json.dump(ROWS, f)
    ^^^^^^^^^
AttributeError: module 'json' has no attribute 'dump'


# The three answers.
#
# 1. FIND. Stage one walked sys.path, and sys.path[0] -- your own folder --
#    answered "yes, I have json" before the standard library was ever asked.
#
# 2. Because the import SUCCEEDED. Nothing was missing, so nothing raised at
#    import time. Python found a module named json, ran it, and bound it. The
#    failure came later, on attribute lookup, when the module it handed you
#    turned out to have as_rows and no dump. ModuleNotFoundError means "no file
#    answered"; AttributeError here means "the wrong file answered".
#
# 3. One line, and it ends the argument:


# ------------- $ python -c "import json; print(json.__file__)" -------------
C:\tmp\report\json.py


# ------------- the fix: rename YOUR file -------------
$ mv json.py rows.py                  # ren json.py rows.py on Windows

$ python save_report.py
wrote report.json

$ cat report.json
[{"track": "Titanium", "plays": 12}]

$ python -c "import os, json; print('/'.join(json.__file__.replace(os.sep,'/').split('/')[-2:]))"
json/__init__.py                      # the real one, at last


# ------------- the corner nobody checks -------------
$ ls __pycache__
json.cpython-312.pyc                  # your old file's bytecode, still there

# Can it still be imported? No -- and the proof is one call:

$ python -c "import importlib.util as u, os; print(os.path.basename(u.find_spec('json').origin))"
__init__.py

# Since Python 3.2 a .pyc inside __pycache__ is only ever consulted as the cache
# of a source file sitting beside it. Delete the .py and the .pyc is inert: the
# finder is looking for json.py, not for json.cpython-312.pyc. So the leftover is
# harmless clutter -- delete the folder if it bothers you, and nothing changes.


# The one honest sentence to take away:
#   sys.path[0] is your folder, so naming a file is choosing which import wins.
#   Check the name against sys.stdlib_module_names BEFORE you save, not after.

The second extension point, path hooks, is the scalpel. Put a callable on sys.path_hooks; each hook is handed a sys.path entry string and either returns a path entry finder for it or raises ImportError to decline. This only affects things reachable through sys.path entries — which is precisely how zipimport works. The zipimporter is a path hook that claims .zip entries, and the sys.path_importer_cache memoises the mapping so the archive is opened once. That is the full explanation of a trick that has always looked like sorcery:

>>> import sys
>>> sys.path.insert(0, "libs.zip")   # a path to an ARCHIVE, not a folder
>>> import packed_module            # read straight out of the zip
>>> sys.path_importer_cache["libs.zip"]
<zipimporter object "libs.zip">      # the path hook that claimed it

Hold the precedence in your head as one ordered pipeline. The sys.modules cache comes first. Then come the sys.meta_path finders in order — yours can go at the front to override, or at the back to be a fallback. PathFinder is itself just the last meta path finder, and it fans out to sys.path_hooks per entry. For correctness a well-behaved loader sets module.__spec__, __name__, __loader__, and __file__ (or leaves __file__ unset for non-file sources), and executes into module.__dict__ via exec_module. Meanwhile the machinery, not your loader, handles inserting into sys.modules and unwinding on error.

Traceback (most recent call last):
  File "C:\tmp\mono\app.py", line 1, in <module>
    from company.auth import login
ModuleNotFoundError: No module named 'company.auth'; 'company' is not a package

Traceback (most recent call last):
  File "C:\tmp\shop\checkout.py", line 1, in <module>
    import cart
  File "C:\tmp\shop\cart.py", line 2, in <module>
    import pricing
  File "C:\tmp\shop\pricing.py", line 4, in <module>
    RATE = cart.TAX
           ^^^^^^^^
AttributeError: partially initialized module 'cart' has no attribute 'TAX' (most likely due to a circular import)

Traceback (most recent call last):
  File "C:\tmp\pycbox\player.py", line 2, in <module>
    import beat
ImportError: bad magic number in 'beat': b'\xa7\r\r\n'

# ---------- studio/metrics.py ----------
# metrics.py -- the shared maths
def mean(values):
    return sum(values) / len(values)


# ---------- studio/run_report.py ----------
# run_report.py -- lives beside metrics.py
import metrics

print("mean:", metrics.mean([200, 245, 320]))


# ---------- studio/notebooks/explore.py ----------
# notebooks/explore.py -- the SAME import line, one folder down
import os
import sys

print("sys.path[0] :", os.path.basename(sys.path[0]) or "(cwd)")

import metrics

print("mean        :", metrics.mean([200, 245, 320]))


# ------------- all three, run from inside studio/ -------------
$ python run_report.py
mean: 255.0

$ python notebooks/explore.py
sys.path[0] : notebooks
Traceback (most recent call last):
  File "C:\tmp\studio\notebooks\explore.py", line 7, in <module>
    import metrics
ModuleNotFoundError: No module named 'metrics'

$ python -m notebooks.explore
sys.path[0] : studio
mean        : 255.0


# The explanation, in one sentence:
#   sys.path[0] is the folder of the SCRIPT you launched -- notebooks -- and
#   metrics.py is not in notebooks, so the search never reaches it.
#
# Which is also why `cd notebooks` first changes nothing: the script form does
# not care where you are standing, only where the file is. Only -m is different,
# because -m puts the CURRENT DIRECTORY at index 0 instead -- studio, where
# metrics.py actually lives. Same file, same import line, different first entry.
#
# And -m works with no __init__.py because notebooks/ resolves as a namespace
# package (section 5) -- enough to be imported through, which is all -m needs.


# ------------- fix B: anchored to the FILE -------------
# in notebooks/explore.py, above `import metrics`:
from pathlib import Path

sys.path.append(str(Path(__file__).resolve().parent.parent))   # studio/

$ python notebooks/explore.py                    # run from studio
sys.path[0] : notebooks
mean        : 255.0
$ cd .. && python studio/notebooks/explore.py   # run from one folder up
sys.path[0] : notebooks
mean        : 255.0                              # still fine: anchored to __file__


# ------------- fix C: anchored to nothing -------------
# ---------- notebooks/explore.py, fix C ----------
# notebooks/explore.py -- the SAME import line, one folder down
import os
import sys

print("sys.path[0] :", os.path.basename(sys.path[0]) or "(cwd)")

sys.path.append("..")            # relative to the WORKING DIRECTORY
import metrics

print("mean        :", metrics.mean([200, 245, 320]))


$ cd notebooks
$ python explore.py
sys.path[0] : notebooks
mean        : 255.0                        # works: ".." happens to BE studio

$ cd ..
$ python notebooks/explore.py
sys.path[0] : notebooks
Traceback (most recent call last):
  File "C:\tmp\studio\notebooks\explore.py", line 8, in <module>
    import metrics
ModuleNotFoundError: No module named 'metrics'

# Same file. Same line. The only thing that changed is where you were standing
# when you pressed Enter -- because ".." is resolved against the CWD, and the
# CWD is not a property of your project.


# Ranked, with reasons:
#   A (python -m from the project root) -- ship this. Zero code, and it is the
#     same command your test runner and your CI already use.
#   B (append Path(__file__)...) -- acceptable when a file must be runnable on
#     its own. It survives a cd, but it hides a real layout problem in a
#     script that now edits interpreter state before it does any work.
#   C (append "..") -- never. It depends on where the human was standing, which
#     is the one thing you cannot put in a repository.
#
# The one honest sentence to take away:
#   "It works on my machine" is usually "sys.path[0] was different on my machine."

You can now see, extend, and even hijack the machine that turns disk bytes into live modules. Next: once a module is live, what happens when your program needs to send its objects to another process — where nothing but bytes can cross? Serialization and the pickle pipe →