28Multiprocessing & IPC — real parallelism across the boundary

In Chapter 27 we made one core do the work of thousands. A single event loop juggled suspended frames, never blocking, never idle. But every one of those frames still ran on one core. Async never touches the oldest law of the interpreter you've lived inside since Volume 1: one interpreter, one lock, one core allowed to run Python at a time. In this chapter we finally break it and buy the other seven cores back. Here's the plan. First we feel the wall — eight cores spun up, seven sitting flat while a pure-compute job crawls on the one. Then we understand exactly why the wall is there. Then we build the machine that gets past it, handing the work to a second interpreter living in a second process. And the whole way through, we keep asking the one question that governs every border in this volume: if two processes share no memory, how does a single value ever cross from one to the other? By the end we've launched real workers onto real cores. We've watched a job finish several times faster instead of mysteriously slower. And we've counted the exact toll the boundary charges at the gate. Because none of it is free: what crosses is always bytes, never objects, flattened on one side and rebuilt on the other.

01The need: the GIL caps you to one core

Let's start with the experiment that should have worked. You've got a pure-Python function that just grinds — counting primes, hashing, anything that's all CPU and no waiting on the outside world. So you do the obvious thing. You split its range into four chunks, hand each chunk to a thread, and expect a machine with four idle cores to finish in roughly a quarter of the time. Chapter 26 sold you threads for exactly this: more workers, more throughput. Watch what actually happens when you run it.

from concurrent.futures import ThreadPoolExecutor
import time

def count_primes(lo, hi):
    total = 0
    for n in range(lo, hi):
        if n > 1 and all(n % d for d in range(2, int(n**0.5) + 1)):
            total += 1
    return total

if __name__ == "__main__":
    chunks = [(2, 150_000), (150_000, 300_000),
              (300_000, 450_000), (450_000, 600_000)]

    t = time.perf_counter()
    for c in chunks:
        count_primes(*c)
    print("serial:  ", round(time.perf_counter() - t, 2), "s")

    t = time.perf_counter()
    with ThreadPoolExecutor(max_workers=4) as pool:
        list(pool.map(lambda c: count_primes(*c), chunks))
    print("4 threads:", round(time.perf_counter() - t, 2), "s")

On the machine these words were written on, the two lines print almost the same number — the four-thread version is a hair slower. Four workers, four cores, zero speedup. Before you blame the code, understand that no amount of rewriting will fix it, because the ceiling is not in your program. It is one object inside CPython called the Global Interpreter Lock.

The GIL is a single mutex — one lock — and the rule around it is brutally simple. A thread must hold the GIL to execute a single byte of Python bytecode, or to touch any object's reference count. The interpreter's engine is the big evaluation loop called _PyEval_EvalFrameDefault, which Volume 1 chapter 4 introduced. It only turns over while the calling thread owns that lock. So at any instant, across your whole process, exactly one thread is running bytecode. The others are stopped, holding their breath, waiting for the lock. Your four threads do not run side by side. They take turns.

They take fast turns, which is why threads still felt useful for I/O. Every ~5 milliseconds — the value in sys.getswitchinterval() — the running thread voluntarily drops the GIL so another can grab it. It also drops the lock the moment it makes a blocking call like socket.recv or a disk read. That is the whole trick of Chapter 26. While one thread sits blocked on I/O, it is not holding the GIL, so a second thread runs during the wait. Waiting is free to overlap. But CPU work never waits — it always wants the lock. So on a pure-compute job the four threads just pass one core back and forth, and you even pay a small tax for every hand-off. Concurrency, yes; parallelism, no.

Why would the language designers hobble every thread like this? Because of a promise Volume 1 made about memory. Every Python object carries a reference count, ob_refcnt, and the interpreter bumps it up (Py_INCREF) and down (Py_DECREF) constantly — every time a name binds to an object or a frame returns. That increment is not atomic. At the machine level it is read, add one, write back. Say two threads on two cores did that to the same object at the same time. One update would clobber the other, the count would drift, and the object would either be freed while still in use (a crash) or never freed (a leak). The GIL is the shortcut that makes refcounting safe: hold one lock, and no two refcount operations can ever collide. One lock buys memory safety for the entire object model, at the cost of one core.

If one GIL is the ceiling, you need a second GIL, and a second GIL means a second interpreter in a second process. So the next question is the most literal one in the whole chapter: what is a process, and how does Python conjure one? →

02A process is its own interpreter and its own GIL

You have decided you need a second interpreter. But you cannot just ask CPython for one inside your program. The whole point is that this interpreter and its GIL are baked into the running process. To get a second GIL you must get a second process, and that is not a Python idea at all. It is a kernel idea, and it is worth being exact about, because the word "process" is about to carry the entire weight of this chapter.

A process is a container the operating-system kernel builds and owns. Three things live in that container. First, its own virtual address space — a private map, held in the CPU's page tables, from the addresses your program uses to the physical RAM behind them. When Volume 1 printed an object's address like 0x7f9c..., that number was only ever meaningful inside one process's map. Second, its own file-descriptor table — the small integers (0, 1, 2, ...) that name its open files and sockets, from Chapter 15. Third, its own PID, the number the kernel and os.getpid() use to refer to it. And because a process loads its own private copy of the interpreter into that address space, it gets its own eval loop and — the payoff — its own GIL. Two processes, two GILs. Now the OS scheduler is free to place them on two different physical cores. That is genuine parallel bytecode. Watch two PIDs appear:

import os
from multiprocessing import Process

def child():
    print("child  pid", os.getpid(), "parent", os.getppid())

if __name__ == "__main__":
    print("parent pid", os.getpid())
    p = Process(target=child)
    p.start()          # kernel creates a new process here
    p.join()           # wait for it to finish

Two different PIDs print. Two processes, side by side, each with its own lock. But how the kernel made that second process is where things get interesting. It is also where a default silently flipped out from under most programmers. multiprocessing offers a start method, and the two that matter are fork and spawn.

you type
# ---------- skeleton.py ----------
import multiprocessing as mp
import os

def job(label, n):                      # module-level def. the child re-imports it.
    print(f"  {label}: pid {os.getpid()}, sum {sum(range(n))}", flush=True)

if __name__ == "__main__":              # THE GUARD -- non-negotiable
    print(f"parent pid {os.getpid()}, start method {mp.get_start_method()}", flush=True)

    p = mp.Process(target=job, args=("child", 1000), name="counter")
    print(f"before start: alive={p.is_alive()} pid={p.pid} exitcode={p.exitcode}", flush=True)
    p.start()                           # the kernel makes the process HERE
    print(f"after  start: alive={p.is_alive()} pid={p.pid}", flush=True)
    p.join()                            # block until it finishes
    print(f"after  join : alive={p.is_alive()} exitcode={p.exitcode}", flush=True)


# ---------- no_guard.py ----------  the SAME program, guard deleted. run it.
from multiprocessing import Process
import os

def worker():
    print("  child", os.getpid(), flush=True)

print("module body", os.getpid(), flush=True)
p = Process(target=worker)      # <-- at MODULE LEVEL, outside any guard
p.start()
p.join()

you see
$ python skeleton.py
parent pid 59368, start method spawn
before start: alive=False pid=None exitcode=None
after  start: alive=True pid=61464
  child: pid 61464, sum 499500
after  join : alive=False exitcode=0

$ python no_guard.py
module body 54908
module body 55288
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  [... 12 frames, all inside multiprocessing/ and runpy ...]
  File "...\multiprocessing\spawn.py", line 140, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html

fork() is a Unix syscall — the default on Linux — and it is almost magical in its cheapness. It clones the calling process. The child comes into existence with a byte-for-byte copy of the parent's entire address space, the same open file descriptors, the same imported modules, the same values in every global. But the kernel does not actually copy the RAM; that would be enormous. It marks every page copy-on-write: parent and child both point at the same physical pages, flagged read-only, and only when one side writes a page does the kernel duplicate that single page. A fork is therefore near-instant (~1ms) and cheap in memory until you start mutating. The child inherits a live snapshot of the parent for free.

That free snapshot is also a trap. Suppose any other thread in the parent was holding a lock at the instant of the fork — a lock inside the memory allocator, say. The child inherits that lock in its locked state, but not the thread that would have released it. The child can then deadlock the first time it touches the allocator. This is why fork plus threads is unsafe. It is also the concrete reason Apple's system libraries made fork so dangerous on macOS that Python changed the default there to spawn in 3.8. Windows never had fork at all.

spawn is the safe, portable alternative and now the default on macOS and Windows. It shares nothing. The kernel launches a brand-new python executable from scratch (via posix_spawn or Windows CreateProcess). That fresh interpreter boots up empty and re-imports your module to rebuild the target function's definition. Then the parent ships the target (function, args) across a pipe as pickled bytes — the theme of this whole volume, arriving right on cue. It is clean, isolated, and slow: booting an interpreter and re-importing takes tens of milliseconds. The re-import is also why spawn demands the if __name__ == "__main__": guard. The child re-runs your module top to bottom. If your Process(...).start() call sits unguarded at module level, every child re-runs it and spawns its own children — an infinite fork-bomb of respawns that Python detects and stops with a RuntimeError.

# child_worlds.py -- how many times does this file's body run?
import multiprocessing as mp
import os

TALLY = ["module-body"]                      # a module-level global

print(f"MODULE BODY  pid={os.getpid():<6} __name__={__name__!r}", flush=True)


def worker(tag):
    TALLY.append(tag)                        # mutate MY OWN copy of the global
    print(f"  child      pid={os.getpid():<6} TALLY={TALLY}", flush=True)


if __name__ == "__main__":                   # only the PARENT gets past this line
    TALLY.append("parent-edit")              # a mutation made BEFORE any launch
    print(f"PARENT       pid={os.getpid():<6} TALLY={TALLY}", flush=True)
    print(f"start method: {mp.get_start_method()}\n", flush=True)

    kids = [mp.Process(target=worker, args=(f"kid-{i}",)) for i in range(3)]
    for k in kids:
        k.start()
    for k in kids:
        k.join()

    print(f"\nPARENT after join         TALLY={TALLY}", flush=True)

$ python child_worlds.py
MODULE BODY  pid=41964  __name__='__main__'
PARENT       pid=41964  TALLY=['module-body', 'parent-edit']
start method: spawn

MODULE BODY  pid=45984  __name__='__mp_main__'
  child      pid=45984  TALLY=['module-body', 'kid-0']
MODULE BODY  pid=38560  __name__='__mp_main__'
  child      pid=38560  TALLY=['module-body', 'kid-1']
MODULE BODY  pid=45060  __name__='__mp_main__'
  child      pid=45060  TALLY=['module-body', 'kid-2']

PARENT after join         TALLY=['module-body', 'parent-edit']

You now have a second process with a second GIL on a second core. But it came at a price the fork/spawn split already hinted at: the two processes share nothing. Your objects live in one address space; the worker lives in another. So how does a value ever get from here to there? →

03Nothing is shared — everything crosses as bytes

Here is a bug that produces no error, which makes it the worst kind. You build a list, hand it to a worker process, the worker appends to it, you wait for the worker to finish, and then you look at your list back in the parent. It is unchanged. No exception, no warning — just silence, and a result that is flatly wrong.

from multiprocessing import Process

def worker(data):
    data.append(999)          # mutate the list we were handed
    print("child sees:", data)   # [1, 2, 3, 999]

if __name__ == "__main__":
    shared = [1, 2, 3]
    p = Process(target=worker, args=(shared,))
    p.start(); p.join()
    print("parent sees:", shared)  # [1, 2, 3]  — the 999 is gone!

The child clearly saw [1, 2, 3, 999]. The parent sees [1, 2, 3]. Nobody lied and nobody crashed. The hard truth is that the worker never touched your list. It could not have. And once you see why, an entire category of multiprocessing confusion dissolves at once.

Recall from the last section that a Python object lives at a virtual address inside one process's address space. Your list shared sits at some address like 0x7f9c0e21a3c0 in the parent. That number is an index into the parent's page tables — its private map. In the child, the exact same number 0x7f9c0e21a3c0 either points at a completely different physical page or at nothing mapped at all. A pointer is meaningless across the boundary. So you cannot pass a reference to an object between processes. There is no wire on which a pointer means the same thing on both ends. This is the metal underneath Volume 1's whole model of names and objects. Names are cheap because they are just pointers into one address space. And the instant you leave that address space, the pointer is worthless.

you type
# two_worlds.py -- the SAME worker, run as a thread (ch 26) and as a process (ch 28)
import multiprocessing as mp
import threading

HITS = []                       # one module-level list
TOTAL = 0                       # one module-level int


def bump(tag, amount):
    global TOTAL
    HITS.append(tag)
    TOTAL += amount
    print(f"    inside {tag:<8} HITS={HITS} TOTAL={TOTAL}", flush=True)


if __name__ == "__main__":
    t = threading.Thread(target=bump, args=("thread", 100))
    t.start(); t.join()
    print(f"  parent sees          HITS={HITS} TOTAL={TOTAL}\n", flush=True)

    p = mp.Process(target=bump, args=("process", 7))
    p.start(); p.join()
    print(f"  parent sees          HITS={HITS} TOTAL={TOTAL}", flush=True)

you see
$ python two_worlds.py
    inside thread   HITS=['thread'] TOTAL=100
  parent sees          HITS=['thread'] TOTAL=100

    inside process  HITS=['process'] TOTAL=7
  parent sees          HITS=['thread'] TOTAL=100

So multiprocessing does the only thing it can. It does not send the object; it sends the object's value, flattened into a stream of bytes, and rebuilds a brand-new object from those bytes on the far side. The machine that does the flattening is pickle. pickle.dumps(obj) walks the whole object graph — the list, the ints inside it, a dict it might point at — and emits a little bytecode program. That program is a stream of pickle opcodes (protocol 5 is the default since Python 3.8) that, when replayed by pickle.loads, reconstructs an equivalent object. Those bytes then travel through a Pipe. On Unix that is an os.pipe() or socketpair() — file descriptors with a kernel ring buffer between them — and on Windows a named pipe. A write() syscall copies the bytes into kernel space; a read() on the other end copies them out. The child unpickles them into freshly allocated objects with their own addresses and their own refcounts, entirely disconnected from yours.

you type
# pipe_queue.py -- the two mailboxes: many-to-one Queue, one-to-one Pipe
import multiprocessing as mp
import pickle


def producer(q, tag):                       # runs in the CHILD
    for row in ([1, 2, 3], {"tag": tag}, "done-ish"):
        q.put({"from": tag, "payload": row})
    q.put(None)                             # the sentinel. None pickles fine.


def echo(conn):                             # runs in the CHILD
    got = conn.recv()                       # blocks until bytes arrive
    conn.send(("echoed", got, id(got)))
    conn.close()


if __name__ == "__main__":
    q = mp.Queue()                          # a pipe + a feeder thread + a lock
    p = mp.Process(target=producer, args=(q, "worker-A"))
    p.start()
    while (item := q.get()) is not None:    # blocks, unpickles, hands you an object
        print("queue  ->", item)
    p.join()

    here, there = mp.Pipe()                 # two connected endpoints
    c = mp.Process(target=echo, args=(there,))
    c.start()
    sent = [10, 20]
    here.send(sent)
    label, back, child_id = here.recv()
    c.join()
    print("pipe   ->", label, back, "| equal:", back == sent, "| same object:", id(sent) == child_id)

    for what, obj in [("lambda", lambda x: x), ("open file", open(__file__)),
                      ("closure", (lambda k: (lambda v: v + k))(3))]:
        try:
            pickle.dumps(obj); print(f"{what:<10} boarded")
        except Exception as e:
            print(f"{what:<10} {type(e).__name__}: {str(e)[:64]}")

you see
$ python pipe_queue.py
queue  -> {'from': 'worker-A', 'payload': [1, 2, 3]}
queue  -> {'from': 'worker-A', 'payload': {'tag': 'worker-A'}}
queue  -> {'from': 'worker-A', 'payload': 'done-ish'}
pipe   -> echoed [10, 20] | equal: True | same object: False
lambda     PicklingError: Can't pickle <function <lambda> at 0x0000022602175BC0>: attribut
open file  TypeError: cannot pickle 'TextIOWrapper' instances
closure    AttributeError: Can't get local object '<lambda>.<locals>.<lambda>'

Now the silent bug is obvious. The child received a copy, rebuilt from bytes at a new address. It appended 999 to its copy. Your original never moved. And three sharper consequences follow from the same fact, each of which you will hit in real code:

Only picklable things cross. Pickle rebuilds an object by value, so it must know how to serialize it. A top-level function pickles fine — it is stored merely as its qualified name (mymodule.worker) and re-imported in the child, which is exactly why pool targets must be module-level defs. But a lambda, a local closure, an open socket, or a threading lock has no by-value representation, and pickling one raises PicklingError. Mutations do not propagate back. The child mutated its private copy; there is no return path unless you explicitly return the value so it gets pickled home. Identity is lost. id() differs across the boundary and is always fails; only value equality survives, and only if your __eq__ is defined by value.

Copies rebuilt from bytes are correct, but they are not free. Flattening an object, shoving it through a kernel pipe, and inflating it again all cost real wall-clock time — and that cost is why your first parallel program can run slower than the serial one. Time to put a number on the toll. →

04The boundary has a toll

You parallelized a tight loop and it got slower. Not a little slower — noticeably, embarrassingly slower than the plain serial version you were trying to beat. This is the most common first result in all of multiprocessing. And it is not a mystery once you learn to see the two tolls you are paying at the border. Both are real wall-clock time, and both are charged per task. If the work inside a task is smaller than the tolls around it, you lose. Every time.

Toll one: startup. Getting a worker process running is not instant. Under spawn the kernel launches a fresh python executable, that interpreter boots, and then it re-imports your modules. If those modules are numpy or torch, the import alone can cost 100ms or more. Order of tens of milliseconds is typical just to have a worker ready. Under fork it is far cheaper — a copy-on-write clone runs in around a millisecond — but even that is not zero. You pay this once per process you create.

Toll two: serialization and transport. This is the one from the last section, now with a stopwatch on it. Every argument you send a worker is pickled — CPU time roughly proportional to how big and how tangled the object graph is. Then it is copied into the pipe's kernel buffer with a write() syscall, read out the other side, and unpickled into fresh objects (allocation plus graph rebuild). And it happens both ways: the arguments cross out, and the result crosses back. So the round-trip cost of one task is the pickle-and-pipe-and-unpickle of the arguments, plus the same again for the return value.

The pipe itself has a hard limit worth knowing. A Unix pipe's kernel buffer is typically 64KB. Try to push a payload bigger than that, and your write() blocks until the reader drains the far end. So a big argument does not just cost pickle time — it can stall the sender waiting on flow control. This is a direct, physical reason that shipping half a gigabyte through a pipe to a worker is a bad idea, and it foreshadows the escape hatch two sections from now.

Put the pieces together into a cost model you can actually reason with. For one task handed to a worker:

T_parallel  =  startup / N_reused                # pay once, amortized over tasks per worker
            +  (pickle + pipe + unpickle)(args)  # toll out
            +  compute / cores                   # the ONLY part you wanted
            +  (pickle + pipe + unpickle)(result)# toll back

Only one term in that sum — compute / cores — is work you actually care about. Everything else is toll. So the entire question of "will parallelism help?" reduces to one comparison: is compute bigger than the serialization terms wrapped around it? Say you send a worker the task "square this number." The compute is a single multiply — nanoseconds — while the pickle-and-pipe of the argument and the result is microseconds. You pay a thousand-fold overhead to avoid doing a multiply. Do that a hundred thousand times and you have built an extremely elaborate way to run slower than a for loop.

The fix falls straight out of the model. The tolls are fixed per task, so pay them fewer times: make each task big. Instead of a million tasks that each square one number, send a few dozen tasks that each square a huge slice of numbers. The compute per task now dwarfs the toll around it, and the compute / cores term finally dominates. This is called coarse-graining or chunking, and it is the single most important instinct in practical multiprocessing. A million tiny tasks pays the toll a million times; a handful of fat chunks pays it a handful of times.

you type
# tiny_tasks.py -- the toll, with a stopwatch on it
import time
from concurrent.futures import ProcessPoolExecutor

N = 20_000

def square(x):                       # ~60 ns of work
    return x * x

def heavy(x):                        # ~160 us of work -- 2,600x fatter
    return sum(i * i for i in range(x % 50 + 950))

def bench(label, run, base=None):
    t = time.perf_counter(); run(); dt = time.perf_counter() - t
    tail = f"{base / dt:8.2f}x serial" if base else ""
    print(f"  {label:<34}{dt * 1000:9.1f} ms  {tail}", flush=True)
    return dt

if __name__ == "__main__":
    print(f"{N:,} items\n")
    for fn, name in ((square, "square  (~60 ns each)"), (heavy, "heavy   (~160 us each)")):
        print(f"{name}")
        base = bench("serial for-loop", lambda: [fn(x) for x in range(N)])
        with ProcessPoolExecutor(8) as pool:
            list(pool.map(fn, [1] * 8))                      # pay startup first
            bench("8 procs, chunksize=1", lambda: list(pool.map(fn, range(N), chunksize=1)), base)
            bench("8 procs, chunksize=2500", lambda: list(pool.map(fn, range(N), chunksize=2500)), base)
        print()

you see
$ python tiny_tasks.py
20,000 items

square  (~60 ns each)
  serial for-loop                         1.1 ms
  8 procs, chunksize=1                 4677.3 ms      0.00x serial
  8 procs, chunksize=2500                22.1 ms      0.05x serial

heavy   (~160 us each)
  serial for-loop                      3228.1 ms
  8 procs, chunksize=1                11887.7 ms      0.27x serial
  8 procs, chunksize=2500               465.0 ms      6.94x serial

You now know the toll and the fix: pay startup rarely, make tasks fat. But hand-spawning a process per chunk and wiring pipes by hand is miserable. The standard library already assembles all of this into one reusable machine — the pool — which pays startup exactly once and hands you coarse-graining as a single argument. →

05Pools distribute coarse work

You are not going to hand-spawn a process for every chunk of work, pay the startup toll each time, and manually thread pipes between them. The standard library already built that assembly for you, and it is called a process pool. It is the shape of code you will actually deploy for CPU-bound work. And it is where every piece of this chapter finally clicks together into a tool.

The core idea is amortization, straight from the last section. A pool starts N long-lived worker processes once — N defaults to os.cpu_count() — and keeps them alive for the whole job. The startup toll is now paid N times total, not once per task. Between the parent and the workers sit two queues built on pipes, guarded by locks and semaphores: a task queue and a result queue. The parent pickles (function, args) and drops it on the task queue. Each idle worker blocks on a read of that queue. When a task arrives, one worker pulls it, unpickles it, and runs it — each worker on its own core with its own GIL. Then it pickles the return value and pushes it onto the result queue. The parent reads results off and matches each one back to the submission that produced it. That is the entire machine, and the crucial fact is the middle of it: at one instant, all N workers are genuinely running bytecode in parallel. This is the payoff the whole chapter was building toward.

The modern front door is concurrent.futures.ProcessPoolExecutor, the same interface Chapter 26 used for threads. That is the point: you swap one class name and the work now runs on real cores instead of taking turns on one. Here is the prime-counter from section 1, finally getting its speedup:

from concurrent.futures import ProcessPoolExecutor
import os, time

def count_primes(bounds):        # module-level: picklable by name
    lo, hi = bounds
    return sum(1 for n in range(lo, hi)
               if n > 1 and all(n % d for d in range(2, int(n**0.5)+1)))

if __name__ == "__main__":
    chunks = [(i, i + 75_000) for i in range(2, 600_000, 75_000)]
    t = time.perf_counter()
    with ProcessPoolExecutor() as pool:
        total = sum(pool.map(count_primes, chunks))
    print(total, "primes on", os.cpu_count(), "cores in",
          round(time.perf_counter() - t, 2), "s")

On a four-core machine that lands near four times faster than serial — the first honest speedup in the chapter. There are two ways to feed a pool, and the difference matters. submit(fn, arg) returns a Future immediately — a placeholder that gets fulfilled when its result arrives. You can hand a batch of futures to as_completed() to process results in the order they finish. map(fn, iterable, chunksize=K) instead batches K items into each pickled task. And there is your coarse-graining lever from the last section, exposed as a single argument. A well-chosen chunksize is the difference between paying the toll once per item and once per batch.

you type
# pool_shapes.py -- the same four jobs, three ways
import os, time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed

def work(n):                                  # module-level: the ONLY shape a pool accepts
    if n == 3:
        raise ValueError(f"worker refused {n}")
    return sum(i * i for i in range(n * 400_000))

JOBS = [5, 6, 7, 8]

if __name__ == "__main__":
    t = time.perf_counter(); [work(n) for n in JOBS]
    print(f"serial              {time.perf_counter() - t:5.2f} s")

    with ThreadPoolExecutor(4) as tp:
        t = time.perf_counter(); list(tp.map(work, JOBS))
        print(f"ThreadPoolExecutor  {time.perf_counter() - t:5.2f} s   <- same one core")

    with ProcessPoolExecutor(4) as pp:
        list(pp.map(work, [1] * 4))
        t = time.perf_counter(); list(pp.map(work, JOBS))
        print(f"ProcessPoolExecutor {time.perf_counter() - t:5.2f} s   <- four cores")

    with ProcessPoolExecutor(4) as pp:
        futs = {pp.submit(work, n): n for n in [1, 2, 3, 4]}
        for f in as_completed(futs):
            try:
                print(f"  job {futs[f]} -> {f.result()}")
            except ValueError as e:
                print(f"  job {futs[f]} -> raised {type(e).__name__}: {e}")

you see
$ python pool_shapes.py
serial               0.73 s
ThreadPoolExecutor   0.74 s   <- same one core
ProcessPoolExecutor  0.31 s   <- four cores
  job 3 -> raised ValueError: worker refused 3
  job 1 -> 21333253333400000
  job 2 -> 170666346666800000
  job 4 -> 1365332053333600000

Three mechanics will bite you if you do not know them, and all three fall straight out of "everything crosses as bytes." First, the target must be picklable — a module-level def, not a lambda or a nested closure — because it is pickled by qualified name and re-imported inside the worker. Second, results can arrive out of order. map restores submission order for you, but imap_unordered (on the older API) hands them back as they finish — fine, and faster, when you do not care about order. Third, and most usefully, exceptions cross the boundary too. If a worker raises, the exception is pickled, shipped home on the result queue, and re-raised in the parent the moment you read that task's result — traceback and all. Your parent sees the failure as if it had happened locally.

# prime_count.py -- ONE cpu-bound function, three ways to run it
import os
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor


def count_primes(bounds):                    # module-level def -> picklable BY NAME
    lo, hi = bounds
    total = 0
    for n in range(lo, hi):
        if n > 1 and all(n % d for d in range(2, int(n**0.5) + 1)):
            total += 1
    return total


CHUNKS = [(i, i + 75_000) for i in range(2, 600_002, 75_000)]   # 8 fat chunks


def timed(label, run):
    t = time.perf_counter()
    total = run()
    dt = time.perf_counter() - t
    print(f"  {label:<20} {total} primes   {dt:5.2f} s")
    return dt


if __name__ == "__main__":                   # THE GUARD -- spawn re-imports this file
    print(f"{os.cpu_count()} cores, 8 chunks of 75,000, 8 workers\n")

    serial = timed("serial", lambda: sum(count_primes(c) for c in CHUNKS))

    with ThreadPoolExecutor(8) as tp:
        threads = timed("8 threads", lambda: sum(tp.map(count_primes, CHUNKS)))

    with ProcessPoolExecutor(8) as pp:
        list(pp.map(count_primes, [(2, 3)] * 8))          # pay the spawn toll first
        procs = timed("8 processes", lambda: sum(pp.map(count_primes, CHUNKS)))

    print(f"\n  threads   {serial / threads:4.2f}x serial   <- ch 26: no cores gained")
    print(f"  processes {serial / procs:4.2f}x serial   <- ch 28: real cores")

$ python prime_count.py
32 cores, 8 chunks of 75,000, 8 workers

  serial               49098 primes    1.79 s
  8 threads            49098 primes    3.65 s
  8 processes          49098 primes    0.67 s

  threads   0.49x serial   <- ch 26: no cores gained
  processes 2.68x serial   <- ch 28: real cores

from concurrent.futures import ProcessPoolExecutor, as_completed

def risky(n):
    if n == 3: raise ValueError("worker hated 3")
    return n * n

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        futures = [pool.submit(risky, n) for n in range(6)]
        for f in as_completed(futures):
            try:
                print("got", f.result())   # re-raises the worker's exception here
            except ValueError as e:
                print("worker failed:", e)

Pools are the answer for chunky compute over modest data. But there is one case they handle badly: when the data is enormous. Pickling and piping a 500MB array into every worker costs more than the computation. For that, you need the workers to touch the same bytes without ever copying them. →

06Shared memory avoids the copy

Your workers need to crunch a 500-megabyte array. Under the pool model from the last section, every worker gets its own pickled copy. That is half a gigabyte serialized, shoved through a 64KB pipe in thousands of blocking chunks, and rebuilt in the worker — and then the results pickled all the way back. The toll now dwarfs the compute completely. You measured it in section 4 and the model is merciless: when the data term is huge and the compute term is not, serialization is your program. You need a way for two processes to touch the exact same bytes without a single copy crossing a pipe.

The kernel can do exactly this, and it is one of the quietly beautiful tricks in operating systems. Recall that a process's address space is a map from virtual addresses to physical RAM. Nothing stops the kernel from mapping one physical region into two different maps at once. Both processes then have a valid virtual address — likely a different number in each — pointing at the identical physical pages. A write by one process changes the DRAM; the other process, whose map points at that same DRAM, sees the change instantly. No pickle, no pipe, no copy. It is not a message; it is the same memory wearing two names.

multiprocessing.shared_memory.SharedMemory is the Python door to this. Under the hood it is POSIX shm_open plus mmap on Unix, or a named file-mapping on Windows. You allocate a block once, it gets a short string name, and any process that opens that name gets the same physical pages mapped into its own space. The idiomatic move — and the reason this exists — is to lay a NumPy array over the block with np.ndarray(shape, dtype, buffer=shm.buf), so workers read and write scientific data at raw memory speed. And here is the elegant part. The only thing that crosses the process boundary is the tiny name string. The gigabyte never travels. You pickle-and-pipe a dozen bytes and the workers attach to the real data locally.

from multiprocessing import Process
from multiprocessing.shared_memory import SharedMemory
import numpy as np

def square_in_place(name, shape):
    shm = SharedMemory(name=name)            # attach to the SAME physical pages
    arr = np.ndarray(shape, dtype=np.float64, buffer=shm.buf)
    arr[:] = arr * arr                          # write straight into shared RAM
    shm.close()                              # unmap our view; do NOT unlink

if __name__ == "__main__":
    data = np.arange(8, dtype=np.float64)
    shm = SharedMemory(create=True, size=data.nbytes)
    buf = np.ndarray(data.shape, dtype=np.float64, buffer=shm.buf)
    buf[:] = data                               # load once into shared RAM

    p = Process(target=square_in_place, args=(shm.name, data.shape))
    p.start(); p.join()

    print(buf)          # [0. 1. 4. 9. 16. 25. 36. 49.] — no return value crossed
    shm.close()
    shm.unlink()         # exactly ONE process must free the segment

The worker got only a name and a shape. It squared every element, and the parent read the mutated array back with no return value crossing the boundary at all. The mutation was in the shared DRAM the whole time. Compare this to the silent-bug section: there, a mutation vanished because the child had a private copy. Here it persists because there is no copy — both processes are literally writing the same physical bytes.

Power like this comes with three sharp edges, and you must know all of them. First, shared memory is unstructured bytes. No Python objects can live there. A Python object contains pointers into its own address space, which are meaningless in the other process — exactly the problem from section 3. All that can be shared is a raw buffer you interpret with a dtype or a struct format. Second, you own the lifecycle. One process must close() its view, and exactly one must unlink() to release the segment. Forget it and you leak a block of RAM that outlives your program, and Python's resource_tracker will print a warning scolding you for it. Third, and most dangerous, there is no GIL guarding these bytes. The GIL only serializes access within one interpreter; across two processes it does nothing. Two workers writing the same element race at the hardware level, and updates get lost silently.

You fix the race the classic way: a lock. multiprocessing.Lock is itself a shared OS semaphore — a small piece of kernel state both processes can see — and wrapping each write in it serializes access. Or, better when you can arrange it, partition the array so each worker owns a disjoint slice and no two ever touch the same bytes. Then you need no lock at all, and the workers run fully parallel with zero coordination. Both are valid; partitioning is faster when the problem allows it.

# ============ PART A ============

# ---------- collatz_serial.py -- the loop you were given ----------
import time


def steps(n):
    count = 0
    while n != 1:
        n = n // 2 if n % 2 == 0 else 3 * n + 1
        count += 1
    return count


if __name__ == "__main__":
    t = time.perf_counter()
    best = max(range(1, 400_001), key=steps)
    print(f"longest chain below 400,000: {best} ({steps(best)} steps)")
    print(f"serial: {time.perf_counter() - t:.2f} s")


$ python collatz_serial.py
longest chain below 400,000: 230631 (442 steps)
serial: 2.87 s


# ---------- collatz_parallel.py -- the same answer, on every core ----------
import time
from concurrent.futures import ProcessPoolExecutor

LIMIT = 400_001


def steps(n):
    count = 0
    while n != 1:
        n = n // 2 if n % 2 == 0 else 3 * n + 1
        count += 1
    return count


def best_in(bounds):                      # ONE fat task: a whole slice, not one number
    lo, hi = bounds
    winner = max(range(lo, hi), key=steps)
    return steps(winner), winner          # (length, n) -- length FIRST, on purpose


if __name__ == "__main__":
    t = time.perf_counter()
    serial = max(range(1, LIMIT), key=steps)
    st = time.perf_counter() - t
    print(f"serial      : {serial} ({steps(serial)} steps)  {st:5.2f} s")

    chunks = [(lo, min(lo + 25_000, LIMIT)) for lo in range(1, LIMIT, 25_000)]
    with ProcessPoolExecutor(8) as pool:
        list(pool.map(best_in, [(1, 2)] * 8))          # pay the spawn toll first
        t = time.perf_counter()
        length, winner = max(pool.map(best_in, chunks))
        pt = time.perf_counter() - t
    print(f"8 processes : {winner} ({length} steps)  {pt:5.2f} s   {st / pt:.1f}x")
    print(f"tasks: {len(chunks)} chunks of 25,000  (not {LIMIT - 1:,} chunks of 1)")


$ python collatz_parallel.py
serial      : 230631 (442 steps)   2.89 s
8 processes : 230631 (442 steps)   0.90 s   3.2x
tasks: 16 chunks of 25,000  (not 400,000 chunks of 1)


# Q1: why (length, n) and not (n, length)?
#     Because max() on tuples compares element 0 first. Putting the step count
#     there makes a plain max() over the partial results the correct combine
#     step -- no key=, no lambda (which could not cross anyway).
#
# Q2: what breaks with pool.map(steps, range(1, 400_001))?
#     Nothing breaks. It returns the right numbers. It is just catastrophic:
#     400,000 tasks, each one pickle + pipe + syscall + unpickle around a few
#     microseconds of arithmetic. Measured on 100,000 numbers, one per task:


# ---------- collatz_naive.py ----------
import time
from concurrent.futures import ProcessPoolExecutor
from collatz_parallel import steps

if __name__ == "__main__":
    N = 100_000
    t = time.perf_counter(); [steps(n) for n in range(1, N + 1)]
    print(f"serial, {N:,} numbers          {time.perf_counter() - t:6.2f} s", flush=True)
    with ProcessPoolExecutor(8) as pool:
        list(pool.map(steps, [2] * 8))
        t = time.perf_counter(); list(pool.map(steps, range(1, N + 1), chunksize=1))
        print(f"8 procs, one number per task {time.perf_counter() - t:6.2f} s", flush=True)


$ python collatz_naive.py
serial, 100,000 numbers            0.63 s
8 procs, one number per task     208.60 s

#     0.63 s -> 208.60 s. Eight cores, 331x SLOWER, same answer. The fix is one
#     keyword: chunksize=12_500. The lesson is not the keyword, it is that the
#     toll is per task, so you choose how many times to pay it.


# ============ PART B ============

# 1. f = lambda row: row[2]
#    _pickle.PicklingError: Can't pickle <function <lambda> at 0x...>:
#    attribute lookup <lambda> on __main__ failed
#    Pickle stores a function as the pair (module, qualified name) and re-imports
#    it in the child. Every lambda is named "<lambda>", so the name it writes
#    down finds nothing when the child looks it up.
#
# 2. handle = open("data.csv")
#    TypeError: cannot pickle 'TextIOWrapper' instances
#    A file object is a small integer -- a file descriptor -- plus buffers, plus
#    a position, all owned by THIS process's descriptor table (chapter 15).
#    Descriptor 7 in the child is some other file, or nothing. There is no
#    by-value form of "the kernel's handle on my open file". Send the PATH and
#    let the child open it.
#
# 3. add3 = make_adder(3)
#    AttributeError: Can't get local object 'make_adder.<locals>.<lambda>'
#    The qualname says it exactly: make_adder.<locals>.<lambda>. It is a local
#    object, unreachable by import, and it also carries a closure cell holding
#    k=3. Pickle can name module-level things; it cannot name things that exist
#    only inside a call that already returned.
#
# The tie-together:
#    def sort_key(row): return row[2]
#    sort_key.__qualname__  ->  'sort_key'          <- importable. crosses.
#    f.__qualname__         ->  '<lambda>'          <- not importable. does not.
#    Pickle never serialises a function's code. It writes down where to FIND the
#    function, and the child re-imports your module to get it. That is why the
#    rule "pool targets must be module-level defs" is not style advice -- it is
#    the only thing the mechanism can express.
#
#    Fixes: replace the lambda with a def; use functools.partial(f, 3), which
#    pickles because both the function and the argument pickle; or send the
#    filename instead of the file handle.

You have now met all three tools — threads for waiting (Ch 26), async for waiting at massive scale (Ch 27), and processes for real parallel compute (this chapter). Faced with a real workload, how do you pick? The answer is not taste. It is a single causal question. →

07The decision table

Three chapters, three tools, and now a real workload in front of you. The temptation is to choose by vibe — "processes sound powerful, I'll use those." Resist it. The right choice falls out mechanically from a single question you have already been circling, and it is causal, not a lookup table you memorize. The question, from Chapter 26, is this: while this task waits, is the CPU doing work — or just waiting? And when it waits, a second question: how many things wait at once? Every branch below is a direct consequence of one fact you now understand cold — who holds the GIL, and when it is released.

Branch one: the task blocks on I/O, and you have moderate concurrency — dozens to low thousands. Use threads. Here is the mechanism, and it is the reason threads work at all despite section 1: the GIL is released during blocking syscalls. When a thread calls socket.recv or reads a disk, CPython drops the lock before descending into the kernel and reacquires it on return. So while one thread sits parked in a read(), it holds nothing, and another thread runs. The waiting overlaps for free, the OS scheduler preempts for you, and you write ordinary blocking code. The cost is that each thread carries a stack — often a megabyte or more — plus lock-contention overhead. So you top out in the hundreds to low thousands before memory and scheduling overhead bite.

Branch two: the task blocks on I/O, but you have tens of thousands of them at once. Use async. Thirty thousand threads means thirty gigabytes of stacks — a non-starter. Chapter 27's event loop replaces them with one thread sitting on an OS readiness primitive — epoll on Linux, kqueue on BSD/macOS, IOCP on Windows — that can watch tens of thousands of sockets and report which are ready. Each waiting connection costs only a small coroutine object, not a thread stack. But the discipline is absolute. The loop is one thread, so a single blocking call or one CPU-heavy coroutine freezes everything, and every library in the path must be async-aware. It is the highest-scale I/O tool and the least forgiving.

Branch three: the task keeps the CPU busy — it computes, it does not wait. Use processes. This is the whole chapter. CPU work always wants the GIL, so threads serialize it onto one core, and async cannot help at all — there is no I/O wait to hide behind. The only way to run Python bytecode on N cores is N interpreters, which is N processes, each with its own GIL. You pay the startup and serialization toll from section 4, you coarse-grain to amortize it, and you reach for shared memory when the data is large. A pool is the standard vehicle.

you type
# decide.py -- eight jobs, two flavours, three tools. one table settles it.
import asyncio, time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor

def waiting(n):                       # I/O-shaped: the CPU is idle the whole time
    time.sleep(0.25)
    return n

def working(n):                       # CPU-shaped: the CPU is pinned the whole time
    return sum(i * i for i in range(3_000_000))

JOBS = list(range(8))

async def gathered(fn):
    if fn is waiting:
        await asyncio.gather(*(asyncio.sleep(0.25) for _ in JOBS))
    else:
        for n in JOBS:                # CPU work has no await point to yield at
            fn(n)

def clock(run):
    t = time.perf_counter(); run(); return time.perf_counter() - t

def row(fn, label, run):
    base = clock(lambda: [fn(n) for n in JOBS])       # re-time serial RIGHT NOW
    dt = clock(run)
    print(f"    {label:<14}{base:6.2f}s{dt:9.2f}s{base / dt:8.1f}x", flush=True)

if __name__ == "__main__":
    for fn, title in ((waiting, "WAITING  8 x sleep(0.25)"),
                      (working, "WORKING  8 x 3M multiplies")):
        print(f"{title:<30}{'serial':>10}{'tool':>9}{'gain':>9}")
        with ThreadPoolExecutor(8) as tp:
            row(fn, "8 threads", lambda: list(tp.map(fn, JOBS)))
        row(fn, "async gather", lambda: asyncio.run(gathered(fn)))
        with ProcessPoolExecutor(8) as pp:
            list(pp.map(fn, JOBS))                    # pay the spawn toll first
            row(fn, "8 processes", lambda: list(pp.map(fn, JOBS)))
        print()

you see
$ python decide.py
WAITING  8 x sleep(0.25)          serial     tool     gain
    8 threads       2.00s     0.25s     7.9x
    async gather    2.00s     0.26s     7.6x
    8 processes     2.00s     0.26s     7.8x

WORKING  8 x 3M multiplies        serial     tool     gain
    8 threads       6.27s     6.15s     1.0x
    async gather    6.22s     5.97s     1.0x
    8 processes     6.03s     0.78s     7.7x

Real programs are rarely one pure kind, and the framework anticipates the mixes. Two hybrids are worth committing to memory, and both keep the event loop honest:

import asyncio
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor

def crunch(n):              # CPU-bound — would freeze the loop if run inline
    return sum(i*i for i in range(n))

async def main():
    loop = asyncio.get_running_loop()
    with ProcessPoolExecutor() as pool:
        # offload CPU work to a real core; the loop stays responsive
        result = await loop.run_in_executor(pool, crunch, 10_000_000)
    print("crunched:", result)

asyncio.run(main())

When CPU work shows up inside an async application, do not run it inline — it would stall the loop and every other coroutine with it. Hand it to a ProcessPoolExecutor via loop.run_in_executor, and the loop keeps servicing its other sockets while a separate core grinds. Symmetrically, when you must call a blocking library that has no async version, bridge it through a ThreadPoolExecutor the same way. The blocking call parks in a thread (dropping the GIL), and the loop stays free. The event loop stays single-threaded and responsive; the heavy or blocking work happens elsewhere and reports back through an awaitable.

# ---------- SCENARIO 1: 40 URLs, ~300 ms of network wait each ----------
# TOOL       threads -- ThreadPoolExecutor(max_workers=16)
# MECHANISM  CPython releases the GIL before descending into the kernel for
#            socket.recv, and reacquires it on return. So 16 threads park in
#            16 blocking reads at once and the waits overlap. 40 x 300 ms of
#            waiting collapses toward 300 ms x ceil(40/16).
# NOT async  It works and scales further, but requests is a blocking library.
#            You would have to rewrite against httpx/aiohttp for 40 URLs.
# NOT procs  Eight interpreters booting to run code that does nothing but wait.
#            You pay spawn + pickle to buy zero cores you were not using.


# ---------- SCENARIO 2: 2,000 JPEGs, ~80 ms of pure-Python pixel work ----------
# TOOL       processes -- ProcessPoolExecutor(), map(..., chunksize=~30)
# MECHANISM  Pixel loops in Python need the GIL on every bytecode, so threads
#            serialise onto one core. N processes means N interpreters, N GILs,
#            N cores. 80 ms per item is far above the ~200 us per-task toll, so
#            even chunksize=1 would win; chunking just removes the last of it.
# NOT threads  The flat line from chapter 26: 8 workers, same wall clock.
# NOT async    Nothing to await. One loop thread computing is a plain for-loop.
# CAVEAT     If the resize is actually Pillow doing the work in C, Pillow may
#            release the GIL itself -- then threads are enough. Measure first.


# ---------- SCENARIO 3: 30,000 idle WebSocket connections ----------
# TOOL       asyncio
# MECHANISM  Each waiting connection is a suspended coroutine -- a frame on the
#            heap, a few hundred bytes -- not an OS thread with a stack. One
#            epoll/IOCP call reports which of the 30,000 sockets are ready, in
#            time that does not grow with the count.
# NOT threads  30,000 thread stacks is tens of gigabytes reserved and a
#            scheduler asked to manage 30,000 runnable entities. It falls over
#            long before the network does.
# NOT procs  Same objection, worse: 30,000 interpreters.


# ---------- SCENARIO 4: one 4 s transform over a 900 MB NumPy array ----------
# TOOL       neither, at first -- stay in one process and let NumPy do it.
# MECHANISM  A NumPy ufunc drops the GIL and runs a C loop; some builds thread
#            it internally. There is no Python bytecode in the hot path, so the
#            GIL is not the ceiling and there is nothing for a pool to fix.
# IF you must parallelise it: multiprocessing.shared_memory, and pass the
#            NAME. Pickling 900 MB into each worker costs more than the 4 s of
#            maths -- and a 64 KB pipe would carry it in ~14,000 blocking
#            writes. Section 6 exists for exactly this case.
# NOT a plain pool  pool.map(fn, big_array) pickles the array per task.


# ---------- THE THREE CROSS-CUTTING QUESTIONS ----------
# Q1  Which flips at 200 us per item?  Scenario 2.
#     At 80 ms the toll is 0.25% of the work. At 200 us it is comparable to the
#     work, so 2,000 tiny tasks lose to a for-loop -- section 4, measured. The
#     fix is chunking, not more cores: chunksize=100 makes each TASK 20 ms
#     again. Scenarios 1 and 3 do not flip; they are waiting, not computing.
#
# Q2  Which flips if the library releases the GIL?  Scenario 2 (and 4).
#     If the resize is C code that drops the lock, threads run it truly in
#     parallel with no pickling and no spawn -- strictly better than processes.
#     This is the one case where "CPU-bound" does not imply "use processes".
#
# Q3  Shared running total in scenario 2 -- simplest correct fix?
#     RETURN the number and sum in the parent:
#         total = sum(pool.map(resize_one, paths, chunksize=30))
#     Not multiprocessing.Lock, for two reasons. First, a lock protects a shared
#     object and there is no shared object -- each worker's global is its own,
#     so the lock would guard four unrelated integers (section 3). You would
#     need Value/SharedMemory as well, and then the lock. Second, even done
#     correctly it serialises the workers at exactly the moment you paid to run
#     them in parallel. Returning costs one small pickle per task and the
#     parent's sum is free. Reach for a lock only when the shared thing is real
#     and the sharing is unavoidable.

You have crossed every boundary this volume set out to cross: the disk, other encodings, other people's code, future-you, and now other cores and other processes — each one governed by the same law, that only bytes cross a boundary, never objects. The machine that flattens objects to a stream and rebuilds them faithfully is the through-line of the whole volume, and you have now seen its most demanding form: the pickle pipe that lets eight cores finally think at once. →