22Environments & dependencies — venv, pip, reproducibility
In Chapter 21 we organized our own code into packages. We nested modules into a clean import tree we controlled end to end. In this chapter that tidy process has to leave home and be useful. And useful means running thousands of files we did not write: other people's libraries, each pinned to versions that quietly fight one another. Here's the plan. First we'll watch that fight break out on a single shared site-packages. Then we'll build the one thing that ends it: the virtual environment. We'll crack open pyvenv.cfg, the three-line contract that turns one shared interpreter into an isolated one. We'll watch pip unzip a wheel into a folder and write the receipt that lets it uninstall the exact same files back out. We'll watch its resolver search a space of version numbers the way a SAT solver hunts a satisfying assignment. And we'll climb the reproducibility ladder from a loose name up to a hash-verified lockfile. The whole way through we keep asking the one question that actually matters — what actually crosses the boundary into your project, and how do we make it the same bytes every single time? By the end, a venv has stopped being a spell you chant. It becomes what it plainly is: a directory of inert files and a PATH string we reordered ourselves. No service, no magic — just the machine doing exactly what we told it to.
01The need: global site-packages starts a version war
Let's start where the trouble is born — a tool that just works. You type pip install requests, a second passes, and import requests answers, so you keep going. Weeks later you spin up a second project that wants a newer requests, and typing pip install requests again also says done. Nothing warned you, nothing asked. But now go back to the first project and run it, and watch it die deep inside a library you never touched: ImportError: cannot import name ... from urllib3. You edited none of A's files and changed nothing in A, yet A is broken.
Here's the collision, concretely. Project A needs requests 2.20, and requests 2.20 pins its own dependency: urllib3<1.25. Project B needs requests 2.31, which demands urllib3>=1.26. Two projects, two flatly incompatible requirements for a third library neither of them names out loud. The second install didn't sit politely beside the first — it overwrote it. Because on the whole machine there is exactly one directory named requests, and exactly one named urllib3, and an install writes into that one directory.
Put the two demands on a number line and the impossibility is arithmetic, not opinion. requests 2.20 pins urllib3<1.25, so its allowed versions stop at 1.24. requests 2.31 pins urllib3>=1.26, so its allowed versions start at 1.26. The first set tops out below 1.25; the second starts at or above 1.26. No single release is both under 1.25 and at least 1.26 — the gap at 1.25 belongs to neither. One folder, one version, two ranges that share no overlap: something has to lose.
To see why, watch where imports actually look. When you write import requests, Python doesn't scan your disk but walks one specific ordered list of directories called sys.path. It reads that list left to right, binding the first requests/__init__.py it finds. One entry on that list is site-packages: the single folder where installed third-party code lives.
>>> import sys, requests
>>> requests.__file__
'/usr/lib/python3.11/site-packages/requests/__init__.py'
>>> for p in sys.path: print(p)
/home/you/project
/usr/lib/python3.11
/usr/lib/python3.11/site-packages # <- requests was found HERE, the first hitLook hard at what that resolution is. Import is name-only. The string "requests" is the entire key; there is no version anywhere in the lookup. The filesystem has no slot for "requests 2.20" sitting next to "requests 2.31" — a directory is named requests or it is not. So site-packages is, precisely, a key/value store with one physical slot per top-level import name. And pip install is a mutating write into that store: to put 2.31 in, it first deletes the files 2.20 recorded, then unpacks the new files in their place. There is never an instant where both versions coexist. The store simply has no room for two.
This is why no amount of care saves you: it is not that pip is careless or that you forgot a flag. The data model itself has nowhere to put a second version. One hole, and you are trying to seat two pigeons. One global mutable directory, N projects, mutually incompatible pins — the shape guarantees a loser. On a system Python the wound goes deeper, because there site-packages is root-owned. The operating system's own tooling imports out of it — apt, dnf, the package manager that installs your entire OS. A stray sudo pip install that bumps a shared library can leave the machine unable to manage its own packages, and you reach for a Python tool and brick the system's.
requests lived further down the path? It would be dead code. Resolution stops at the first hit, so a copy at path position 3 is unreachable while position 2 has one — Python never even stats it. "Which version am I running?" is answered entirely by path order, not by which was installed last.If one global directory is the disease, the cure writes itself: give every project its own site-packages so the single-slot store is no longer shared. That private directory has a name — a virtual environment — and the next section opens one up to find it is nothing but files. →
02What a venv actually is
People chant "activate your venv" like an incantation, and when python still points at the wrong interpreter after they've said the magic words, they have no model to debug it with. So let's drain the magic out completely. A virtual environment is a directory of files on disk — inert data, not a running service. Nothing about it is live until you act. Once you can picture the files, "activate" stops being a spell and becomes something you can inspect, break, and fix.
python -m venv .venv puts a private copy of Python inside my project, and a copy has its own everything, which is why it can hold different packages from the system one. It costs some disk, and that is a fair price. The reassuring part is what follows from it — the folder is now self-contained, so I can rename it, zip it, commit it, or carry the project to another laptop, and my environment travels with my code.> python -m venv .venv > .venv\Scripts\python.exe -c "import os, sys; print(sys.executable); print(os.__file__)" > dir /b .venv\Lib REM ls .venv/lib/python3.12 on macOS / Linux
C:\tmp\jukebox-fresh\.venv\Scripts\python.exe C:\Users\ajair\anaconda3\Lib\os.py site-packages
.venv — line one spells out its full path — and the very first module it imported, os, answered from C:\Users\ajair\anaconda3, a directory the venv does not own. Then dir /b .venv\Lib prints one word. That is the entire library tree of a virtual environment: a single empty site-packages, with no os.py, no json, no collections — one entry where the base install's Lib holds 201. So a venv is a pointer to an interpreter plus its own empty shelf, and the isolation is not a duplicated Python. It is a second shelf hung off the same one. Two honest refinements, because "no copy" is looser than what really happened. Something was written into .venv\Scripts: a python.exe of 258,048 bytes. It is not a copy of the base python.exe, which is 93,184 bytes and hashes to something else entirely — it is a launcher, and its whole job is to read pyvenv.cfg and hand off. On macOS and Linux even that is skipped, and bin/python is usually a symlink: a directory entry carrying no bytes at all. Nor is the folder weightless — this one is 11.2 MB, but 10.4 MB of that is pip's own vendored code, leaving 822 KB for everything Python put there, against 27.5 MB of standard library it declined to copy. The second half of the belief fails for the same reason the first did: pyvenv.cfg records an absolute path to a base install on this disk, so a venv you zip, commit, or carry to another machine is a signpost pointing at an address that is no longer there.Create one with python -m venv .venv, and that command writes a concrete, boring tree:
$ python -m venv .venv
$ ls -F .venv
bin/ # Scripts\ on Windows
include/
lib/
pyvenv.cfg # a 3-line text file — the whole contract (next section)
$ ls -F .venv/bin
activate # an ordinary shell script
pip*
python -> /usr/bin/python3.11 # a SYMLINK to the base interpreter (POSIX)
$ ls .venv/lib/python3.11/site-packages
# ...empty. Just pip and setuptools bookkeeping. Nothing else yet.Read that tree closely. The python inside bin/ is not a fresh copy of Python: on POSIX it is a symlink to the base interpreter you already had, and on Windows it is a tiny launcher stub. There is a pyvenv.cfg text file at the root, and an empty site-packages/ whose emptiness is the whole point of section 1's fix. And critically, nothing is running: there is no daemon, no background process, no environment variable set. It is a folder, and it will sit there unchanged whether you "activate" it or not.
So what does activation do? source .venv/bin/activate runs an ordinary shell script that performs exactly four boring steps. First it saves your current $PATH and prompt into backup variables, then prepends .venv/bin to the front of $PATH. Then it exports VIRTUAL_ENV=/path/to/.venv, and finally tells the shell to rehash its cache of command locations. That is the entire trick.
# before
$ echo $PATH
/usr/bin:/bin
$ source .venv/bin/activate
# after — one directory spliced onto the front
(.venv) $ echo $PATH
/proj/.venv/bin:/usr/bin:/bin
(.venv) $ which python
/proj/.venv/bin/python # the shell scans PATH left-to-right, stops at the first hitAfter activation, typing python makes the shell walk $PATH left to right. It stops at the first directory containing a python. That is now .venv/bin, because you just moved it to the front. The (.venv) in your prompt is cosmetic, painted on by the same script. And deactivate is the mirror image: it restores the saved $PATH and prompt and unsets VIRTUAL_ENV. There is no state to tear down because there was no state — only a string that got reordered and then put back.
Make it concrete with the actual string. Before activation your PATH might read /usr/local/bin:/usr/bin:/bin, so typing python finds /usr/bin/python, the base interpreter. Activation prepends one entry, and PATH becomes /home/you/proj/.venv/bin:/usr/local/bin:/usr/bin:/bin. Now the left-to-right walk hits .venv/bin/python first and stops. Same shell, same search rule, the same directories still behind it. One segment glued to the front changed which python answers. That is the whole mechanism, spelled out in one line of text.
python -m venv .venv — build the shelf, step onto it on Windows and on unix, step back off# ---------- 1. make it. one command, character-identical on every platform ---------- > python -m venv .venv # ---------- 2. read what it wrote. this folder IS the "environment" ---------- > dir /b .venv REM Windows, cmd.exe > dir /b .venv\Scripts > type .venv\pyvenv.cfg > dir /b .venv\Lib\site-packages $ ls .venv # macOS / Linux -- same tree, POSIX names $ ls .venv/bin $ cat .venv/pyvenv.cfg $ ls .venv/lib/python3.12/site-packages # ---------- 3. activate. ONE idea, four spellings ---------- > .venv\Scripts\activate.bat REM cmd.exe > .venv\Scripts\Activate.ps1 REM PowerShell $ source .venv/Scripts/activate # Git Bash on Windows $ source .venv/bin/activate # macOS / Linux <- the line in every tutorial # ---------- 4. prove which python you are now holding ---------- (.venv) > where python REM Windows prints EVERY match, in PATH order (.venv) > python -m pip --version (.venv) $ which python # POSIX prints only the FIRST match # ---------- 5. step back off. nothing to tear down, because nothing was built ---------- (.venv) > deactivate > where python
> python -m venv .venv
<- silence. it wrote a folder. that is the event.
> dir /b .venv
Include
Lib
pyvenv.cfg
Scripts
> dir /b .venv\Scripts
activate
activate.bat
Activate.ps1
deactivate.bat
pip.exe
pip3.12.exe
pip3.exe
python.exe
pythonw.exe
> type .venv\pyvenv.cfg
home = C:\Users\ajair\anaconda3
include-system-site-packages = false
version = 3.12.7
executable = C:\Users\ajair\anaconda3\python.exe
command = C:\Users\ajair\anaconda3\python.exe -m venv C:\tmp\jukebox-project\.venv
> dir /b .venv\Lib\site-packages
pip
pip-24.2.dist-info <- that is ALL. the shelf is empty.
> where python
C:\Users\ajair\anaconda3\python.exe
C:\Users\ajair\AppData\Local\Microsoft\WindowsApps\python.exe
> .venv\Scripts\activate.bat
(.venv) > where python
C:\tmp\jukebox-project\.venv\Scripts\python.exe <- NEW, and FIRST. that is the whole trick.
C:\Users\ajair\anaconda3\python.exe
C:\Users\ajair\AppData\Local\Microsoft\WindowsApps\python.exe
(.venv) > python -m pip --version
pip 24.2 from C:\tmp\jukebox-project\.venv\Lib\site-packages\pip (python 3.12)
(.venv) > deactivate
> where python
C:\Users\ajair\anaconda3\python.exe
C:\Users\ajair\AppData\Local\Microsoft\WindowsApps\python.exe
cmd.exe and Python 3.12.7. Run it yourself and only the paths will change. Start with step 2, because that is where the mystique dies. python -m venv wrote four entries, and the one that matters, site-packages, holds pip and nothing else. No service started. No variable was set. A folder appeared. Now read the two where python blocks against each other, top and bottom. Before activation the shell found two pythons; after it, three — and the venv's copy sits at the front. activate.bat created nothing at all. It spliced one directory onto a string, and the shell's own left-to-right scan did the rest. deactivate handed the string back, and the list is exactly what it was. Look at what .venv\Scripts does not contain, too: a copy of Python. That python.exe is a small launcher pointing at the base install, which is why a venv costs kilobytes rather than the hundred-odd megabytes of a real interpreter. Two honest notes about the platform lines. This machine runs Windows, so source .venv/bin/activate is not a command I executed — I ran the same script under its Windows home, .venv/Scripts/activate, from Git Bash. One script, two addresses. And if PowerShell answers Activate.ps1 cannot be loaded because running scripts is disabled on this system, that is not your typo: the default execution policy is Restricted, and activate.bat is untouched by it..venv/bin/python script.py by its absolute path and it behaves identically to the activated form — same isolation, same packages. The isolation does not live in the activate script; it lives in the interpreter and its pyvenv.cfg (next section). This is why tools, cron jobs, and Docker images skip activation entirely and just spell out the path. "Activate" is a shortcut for humans at a prompt, nothing more.source .venv/bin/activatepython app.pyreorders $PATH, then runs the first `python`
.venv/bin/python app.pynames the exact interpreter — same result, no $PATH change
source and not just run ./activate? Because a script you execute runs in a child shell and dies, taking its PATH edit with it. source runs the lines in your current shell, so the edit survives. Activation working at all depends on it not being its own process.But this raises a sharp question. PATH only decides which python binary runs — and that binary is a symlink to the same base interpreter you already had. So how does it know to use the venv's empty site-packages instead of the base one, while still finding the standard library back in the base install? The isolation can't come from PATH. It has to happen inside the interpreter, at startup. →
03How the interpreter finds its packages
PATH chose which python runs — but that binary is a symlink to the base interpreter, byte-for-byte the same executable you had before you ever made a venv. Run it and it must somehow use the venv's empty site-packages, while still importing the standard library, which physically lives back in the base installation. Two different roots, one shared binary. The isolation cannot come from PATH, and it cannot come from the executable, because they're identical. It has to be computed by the interpreter itself, in the first milliseconds of startup. This is PEP 405, and it is exact.
At startup CPython must compute sys.prefix, the answer to one question: "where do I find the standard library and site-packages?" To do it, the running python looks in its own directory and the parent directory for a file named pyvenv.cfg. That file is the entire contract, and it is three lines:
home = /usr/bin
include-system-site-packages = false
version = 3.11.4If that file is present, the interpreter flips into venv mode. It sets sys.prefix to the venv root — so its site-packages resolves to .venv/lib/python3.11/site-packages, the empty directory. Then it reads the home key, which points back at the base install's binary directory. From home it locates the real installation by finding a landmark module (os.py) and sets sys.base_prefix to that base root. The standard library is then sourced from base_prefix; the third-party site-packages is sourced from prefix. The include-system-site-packages = false key decides whether the base's site-packages is also tacked on — normally not, for clean isolation. Finally site.py runs and assembles the two into sys.path.
Here it is with real values. Say your venv sits at /home/you/proj/.venv and the base install lives under /usr. The home line reads home = /usr/bin, pointing at the base binary. So sys.prefix becomes /home/you/proj/.venv, and site-packages resolves to .venv/lib/python3.11/site-packages, the empty one. But sys.base_prefix stays /usr, so import os loads /usr/lib/python3.11/os.py from the base. Two roots, computed in the first milliseconds, out of three lines of text.
python -m pip — the pip that belongs to this pythonpip is a program PATH happened to find; -m is a pip the interpreter owns — only one of them can be wrong-mRuns a module as a program using the interpreter on the left. Naming the python chooses the pip; there is no second decision.pipAn executable the shell finds by scanning PATH. Which python it serves is decided by path order, not by you.pip.exe has one interpreter path baked in when it was written. It cannot install anywhere else, ever.== vs >=== names one release. >= floats forward on every install, and neither one pins the transitive graph.sys.executableThe absolute path of the running interpreter. The one witness that cannot be confused by PATH, aliases, or shells.sys.prefix != sys.base_prefixPEP 405's own test, and pip's. True inside a venv even when VIRTUAL_ENV is unset.you type
# ---------- whichpython.py -- ask the interpreter that is ACTUALLY running ----------
import os
import sys
import sysconfig
print("executable :", sys.executable)
print("prefix :", sys.prefix)
print("base_prefix :", sys.base_prefix)
print("in a venv :", sys.prefix != sys.base_prefix)
print("site-packages :", sysconfig.get_paths()["purelib"])
print("VIRTUAL_ENV :", os.environ.get("VIRTUAL_ENV", "<unset>"))
# ---------- run it with BOTH interpreters. nothing is activated. ----------
> python whichpython.py
> .venv\Scripts\python.exe whichpython.py
> python -m pip --version
> .venv\Scripts\python.exe -m pip --versionyou see
> python whichpython.py
executable : C:\Users\ajair\anaconda3\python.exe
prefix : C:\Users\ajair\anaconda3
base_prefix : C:\Users\ajair\anaconda3
in a venv : False
site-packages : C:\Users\ajair\anaconda3\Lib\site-packages
VIRTUAL_ENV : <unset>
> .venv\Scripts\python.exe whichpython.py
executable : C:\tmp\jukebox-project\.venv\Scripts\python.exe
prefix : C:\tmp\jukebox-project\.venv
base_prefix : C:\Users\ajair\anaconda3
in a venv : True <- isolated...
site-packages : C:\tmp\jukebox-project\.venv\Lib\site-packages
VIRTUAL_ENV : <unset> <- ...with the variable never set
> python -m pip --version
pip 24.2 from C:\Users\ajair\anaconda3\Lib\site-packages\pip (python 3.12)
> .venv\Scripts\python.exe -m pip --version
pip 24.2 from C:\tmp\jukebox-project\.venv\Lib\site-packages\pip (python 3.12)- Read run 2 twice:
in a venv : TruebesideVIRTUAL_ENV : <unset>. The isolation came frompyvenv.cfg; the variable is a signpost. where pythonon Windows prints every match. Two before activation, three after — and only the order changed.pip --versionprints the pip and its python. When a package vanishes, that is your first command, not your last.pip install Xandpython -m pip install Xagree only whenPATHagrees.-mdeletes the word "only".- PowerShell may refuse
Activate.ps1withrunning scripts is disabled on this system. The default policy isRestricted;activate.batignores it. sudo pip installis this same mistake wearing root. It writes into the OS's own site-packages — the oneaptanddnfimport from.
>>> import sys
>>> sys.prefix # where site-packages comes from
'/proj/.venv'
>>> sys.base_prefix # where the stdlib comes from
'/usr'
>>> # outside a venv these two are EQUAL. Inside one, they differ.So the final sys.path is a two-color stack. The standard-library entries come from base_prefix, the shared base install, and a single third-party entry comes from prefix: the venv's own site-packages. The base's site-packages is skipped, and that is how one shared binary serves an isolated environment. The stdlib is borrowed from home, and the packages are private.
rich, and the one command that separates them# ---------- whererich.py ----------
# whererich.py -- one question, asked of whichever python is running me.
import sys
from importlib.metadata import version
import rich
print("python :", sys.executable)
print("in venv:", sys.prefix != sys.base_prefix)
print("rich :", version("rich"))
print("from :", rich.__file__)
# ---------- run the SAME FILE with two interpreters. nothing activated. ----------
> python whererich.py
> .venv\Scripts\python.exe whererich.py
# ---------- now stage the bug itself, with two throwaway environments ----------
# .other plays the part of "your global python". I will not vandalise a real one
# to make a point, so both of these are disposable venvs in C:\tmp\wrongpip.
> python -m venv .venv
> python -m venv .other
> .other\Scripts\pip.exe install humanize REM the WRONG pip -- and it SUCCEEDS
> .other\Scripts\pip.exe list
> .venv\Scripts\python.exe -c "import humanize" REM the python you actually run
> .other\Scripts\pip.exe --version REM ask each one who it serves
> .venv\Scripts\python.exe -m pip --version
> .venv\Scripts\python.exe -m pip install humanize REM THE FIX
> .venv\Scripts\python.exe -c "import humanize; print(humanize.__file__)\"
> python whererich.py python : C:\Users\ajair\anaconda3\python.exe in venv: False rich : 13.7.1 <- version A from : C:\Users\ajair\anaconda3\Lib\site-packages\rich\__init__.py > .venv\Scripts\python.exe whererich.py python : C:\tmp\jukebox-project\.venv\Scripts\python.exe in venv: True rich : 13.7.0 <- version B, same machine, same minute from : C:\tmp\jukebox-project\.venv\Lib\site-packages\rich\__init__.py > .other\Scripts\pip.exe install humanize Collecting humanize Using cached humanize-4.16.0-py3-none-any.whl.metadata (8.0 kB) Using cached humanize-4.16.0-py3-none-any.whl (137 kB) Installing collected packages: humanize Successfully installed humanize-4.16.0 <- it worked. it says so. it is telling the truth. > .other\Scripts\pip.exe list Package Version -------- ------- humanize 4.16.0 pip 24.2 > .venv\Scripts\python.exe -c "import humanize" Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'humanize' > .other\Scripts\pip.exe --version pip 24.2 from C:\tmp\wrongpip\.other\Lib\site-packages\pip (python 3.12) > .venv\Scripts\python.exe -m pip --version pip 24.2 from C:\tmp\wrongpip\.venv\Lib\site-packages\pip (python 3.12) > .venv\Scripts\python.exe -m pip install humanize Collecting humanize Using cached humanize-4.16.0-py3-none-any.whl.metadata (8.0 kB) Using cached humanize-4.16.0-py3-none-any.whl (137 kB) Installing collected packages: humanize Successfully installed humanize-4.16.0 > .venv\Scripts\python.exe -c "import humanize; print(humanize.__file__)" C:\tmp\wrongpip\.venv\Lib\site-packages\humanize\__init__.py
whererich.py, run it twice, and read the two blocks as a pair. Same file, same second, same laptop — and rich answers 13.7.1 once and 13.7.0 the other time. That is Section 1's version war sitting quietly on one disk, no longer a war, because there are now two shelves instead of one. The from line names the file that won each lookup, and neither python can see the other's copy. Then read the staged bug, which is the single most common Python support ticket in the world. pip install humanize printed Successfully installed and it was not lying — it installed the package perfectly, into .other. The interpreter you then ran was .venv, and it raised ModuleNotFoundError, also perfectly correctly. Nothing malfunctioned. Two programs answered two different questions, and only you assumed they were the same question. The diagnosis is two lines, and it fits in the width of a terminal: run pip --version, run python -m pip --version, and compare the paths they print. Here they differ at the fourth segment, .other against .venv, and that mismatch is the bug — the whole of it. The fix follows from the diagnosis rather than from folklore. Stop letting PATH choose your installer: name the interpreter and let -m derive its pip. One honest note on where these ran. The two whererich.py runs are from C:\tmp\jukebox-project; the staged bug is from C:\tmp\wrongpip, where .other is a throwaway venv standing in for a system Python. Everything printed above is captured output, and no real system interpreter was written to.pip install requests printed Successfully installed requests-2.32.3. Then python app.py died with ModuleNotFoundError: No module named 'requests'. I ran it twice." B. "My prompt says (.venv), so I am definitely in the environment, but import requests gives me 2.20 and my requirements.txt pins 2.32." C. "pip install requests raised PermissionError: [Errno 13], so I ran sudo pip install requests and it worked fine. Two days later apt stopped working." Second, for each one, write down the broken assumption before you write a command. Not the fix — the belief. In A, what did the reporter assume about the relationship between two executables? In B, what did they assume the (.venv) prefix is? In C, what did they assume a permission error means? Say each one in a single sentence; a fix you cannot justify will be re-broken next week. Third, prove each diagnosis with two commands, then fix it with one. The two commands are the same pair every time, and they must be commands that ask the machine, never the prompt. Write out the exact output you would expect to see if your diagnosis is right — specifically, which part of the two paths differs. Then give the one-line fix. For C, name what sudo actually did to the machine, and which two programs on a Linux box import out of the directory it wrote into. One hint and no more: in B, run deactivate and re-activate before you conclude anything, and note that your shell caches command locations.show the solution
# ===================================================================== # THE ONE-COMMAND DIAGNOSIS -- run this before theorising, every time # ===================================================================== > pip --version pip 24.2 from C:\Users\ajair\anaconda3\Lib\site-packages\pip (python 3.12) > python -m pip --version pip 24.2 from C:\tmp\jukebox-project\.venv\Lib\site-packages\pip (python 3.12) # ^^^^^^^^^^^^^^^^^^^ different roots -> two environments -> the bug # --------------------------------------------------------------------- # SCENARIO A -- "pip said Successfully installed, python says No module" # --------------------------------------------------------------------- # Broken assumption: that `pip` and `python` are two halves of one tool. # They are two independent executables, each resolved by PATH order. # `pip` wrote into the environment ITS shim points at; `python` read from # the environment PATH gave YOU. Different roots, so the import misses. # # Prove it: > pip --version # names pip's python > python -c "import sys; print(sys.executable)" # # Fix: > python -m pip install requests # -m derives pip FROM the python you named # --------------------------------------------------------------------- # SCENARIO B -- "(.venv) is in my prompt but the wrong version imports" # --------------------------------------------------------------------- # Broken assumption: that the (.venv) prefix is a fact about the interpreter. # It is a string the activate script painted onto PS1. Nothing reads it back. # Either the shell cached the old python (rehash) or the file was installed # by a different pip before you activated. # # Prove it -- ask the interpreter, never the prompt: > python -c "import sys; print(sys.executable, sys.prefix != sys.base_prefix)" > python -c "import requests; print(requests.__file__)" # # Fix: > deactivate # then re-activate, so PATH and the hash cache agree > .venv\Scripts\activate.bat > python -m pip install requests # --------------------------------------------------------------------- # SCENARIO C -- "sudo pip install fixed the permission error" # --------------------------------------------------------------------- # Broken assumption: that a permission error means you need more privilege. # It means you are writing to the WRONG site-packages -- the root-owned one # the operating system imports from. sudo does not fix that; it completes it. # apt and dnf are Python programs importing out of that exact directory. # # Prove it: $ python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])" /usr/lib/python3.11/site-packages # not yours. never was. # # Fix -- never sudo. make a venv and write to a shelf you own: $ python -m venv .venv $ .venv/bin/python -m pip install requests # ===================================================================== # THE RULE, IN ONE LINE # ===================================================================== # `pip install X` asks PATH which installer to run. # `python -m pip install X` tells one interpreter to install into itself. # The second question has exactly one answer. That is the whole reason to prefer it.
sys.prefix != sys.base_prefix is literally how Python — and pip, and every tool — detects that it is running inside a virtual environment. It is not a heuristic; it is the definition. A three-line text file flips one of those two values, and that single inequality is the entire difference between an isolated environment and a shared one.rm .venv/pyvenv.cfg, what happens? The venv's python can no longer find the file that tells it to flip prefixes, so it boots as the base interpreter — sys.prefix snaps back to /usr, and imports start hitting the system's site-packages again. The isolation was that one file the whole time. Delete it and the venv silently un-becomes.The environment is now isolated, and its site-packages is empty. Time to fill it. When you type pip install rich it's importable in a second; pip install numpy sometimes grinds for a minute and fires up a C compiler. Same command, wildly different work — what physically lands in that directory? →
04pip installs wheels into site-packages
pip install rich finishes before you've let go of Enter, and import rich works. pip install numpy sometimes chews for a minute, prints "Building wheel...", and spins up a C compiler. Same command, two utterly different machines behind it. You deserve to know exactly what happens to that empty site-packages directory during an install — and why one case is a file-copy and the other is a build.
A wheel (.whl) is nothing exotic: it is a ZIP archive with a standardized filename and internal layout. The filename alone encodes compatibility, in five dash-separated fields:
Decode a real one: rich-13.7.0-py3-none-any.whl. The five dash-separated fields read left to right. rich is the distribution, 13.7.0 the version, py3 the Python tag (any Python 3), none the ABI tag (no compiled C, so no ABI to match), and any the platform tag (any OS). That none-any ending is the tell: pure python, installs by unzipping. Contrast numpy's numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.whl, whose fields lock it to one interpreter ABI and one OS.
rich-13.7.0-py3-none-any.whl
| | | |__ platform tag: any OS -> pure python
| | |______ abi tag: none
| |__________ python tag: py3 (any Python 3)
|_________________ version
numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.whl
| | |__ platform: Linux x86_64 -> COMPILED binary
| |________ abi: CPython 3.11 ABI
|______________ python: CPython 3.11 onlyInside the ZIP are the package directories plus one <name>-<version>.dist-info/ folder of bookkeeping. That folder holds METADATA (name, version, and the all-important Requires-Dist lines), RECORD (every installed file with its sha256 and byte size), WHEEL, and entry_points.txt. Installing a pure-python wheel is mechanically trivial: you unzip the files into site-packages, write the RECORD, generate console-script shims from entry_points, and optionally byte-compile .pyc files. No build step, and no arbitrary code runs — that is exactly why it is fast, and why it is safe.
>>> import zipfile
>>> z = zipfile.ZipFile("rich-13.7.0-py3-none-any.whl")
>>> z.namelist()[:4]
['rich/__init__.py', 'rich/console.py', 'rich/text.py',
'rich-13.7.0.dist-info/RECORD']
>>> print(z.read("rich-13.7.0.dist-info/RECORD").decode()[:90])
rich/__init__.py,sha256=Q...,6072
rich/console.py,sha256=v...,99123 # path, hash, size — the receiptThe contrast is the sdist (.tar.gz): raw source plus a build backend (setup.py or pyproject.toml). Installing an sdist means pip must first build a wheel. It runs the backend, possibly invokes a C compiler, and links against system libraries. Only then does it unpack the wheel it just made. That is numpy's lost minute: it isn't downloading slowly, it's compiling. And the platform tag told you which path you were on before anything ran. A none-any tag means pure python, just files. A cp311-...-manylinux tag means a precompiled binary for one interpreter ABI and OS.
rich-...-py3-none-any.whlunzip → write RECORD → done
no build, no compiler, no code runs. Fast & safe.
numpy-...-cp311-manylinux.whl or .tar.gzrun build backend → C compiler → wheel → unzip
arbitrary build code, minutes, system libs.
And uninstall is the mirror image, which is why it is reliable: pip reads RECORD and deletes exactly the files it lists — no guessing, no leftovers. So site-packages is demystified entirely: it is nothing but unpacked wheel files plus dist-info bookkeeping. The one-second install and the one-minute compile are the same act: get a wheel, unzip it. They differ only in whether the wheel already existed or had to be built first.
# ---------- the shelf, before you touch it ---------- (.venv) > python -m pip list # ---------- install. two flavours, and the difference is the whole chapter ---------- (.venv) > python -m pip install rich REM "whatever is newest TODAY" (.venv) > python -m pip install rich==13.7.0 REM "that release. forever." # ---------- inspect ---------- (.venv) > python -m pip list (.venv) > python -m pip show rich # ---------- remove ---------- (.venv) > python -m pip uninstall -y rich (.venv) > python -m pip list
(.venv) > python -m pip list Package Version ------- ------- pip 24.2 <- an empty shelf. this is a fresh venv. (.venv) > python -m pip install rich==13.7.0 Collecting rich==13.7.0 Using cached rich-13.7.0-py3-none-any.whl.metadata (18 kB) Collecting markdown-it-py>=2.2.0 (from rich==13.7.0) Using cached markdown_it_py-4.2.0-py3-none-any.whl.metadata (7.4 kB) Collecting pygments<3.0.0,>=2.13.0 (from rich==13.7.0) Using cached pygments-2.20.0-py3-none-any.whl.metadata (2.5 kB) Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich==13.7.0) Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB) Using cached rich-13.7.0-py3-none-any.whl (240 kB) Using cached markdown_it_py-4.2.0-py3-none-any.whl (91 kB) Using cached pygments-2.20.0-py3-none-any.whl (1.2 MB) Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB) Installing collected packages: pygments, mdurl, markdown-it-py, rich Successfully installed markdown-it-py-4.2.0 mdurl-0.1.2 pygments-2.20.0 rich-13.7.0 (.venv) > python -m pip list Package Version -------------- ------- markdown-it-py 4.2.0 <- you asked for ONE name mdurl 0.1.2 pip 24.2 Pygments 2.20.0 rich 13.7.0 (.venv) > python -m pip show rich Name: rich Version: 13.7.0 Summary: Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal Home-page: https://github.com/Textualize/rich Author: Will McGugan Author-email: willmcgugan@gmail.com License: MIT Location: C:\tmp\jukebox-project\.venv\Lib\site-packages Requires: markdown-it-py, pygments <- its Requires-Dist, read off METADATA Required-by: (.venv) > python -m pip uninstall -y rich Found existing installation: rich 13.7.0 Uninstalling rich-13.7.0: Successfully uninstalled rich-13.7.0 (.venv) > python -m pip list Package Version -------------- ------- markdown-it-py 4.2.0 <- still here. uninstall is NOT recursive. mdurl 0.1.2 pip 24.2 Pygments 2.20.0
Collecting line is the resolver reading one package's Requires-Dist and narrowing the allowed sets. Only when the last constraint holds does Installing collected packages: appear, and that single line is Section 4's unzip loop running four times. Now count. You typed one name and pip list shows four packages. markdown-it-py, mdurl and Pygments arrived because rich asked for them, and pip show rich prints that request back to you on its Requires line — the same Requires-Dist field the resolver read out of METADATA. The Location line is worth pausing on: it is the venv's site-packages, spelled out, and it is the answer to "where did that actually go?". Then read the uninstall, which is the surprise. rich is gone and its three dependencies are still there. pip removes exactly what RECORD lists for the package you named, and it never guesses that a leftover is unwanted — something else might depend on it. That is why an environment silently accretes packages over months, and why the honest repair is to delete .venv and rebuild it from a requirements file rather than to tidy it by hand. Which is precisely the file the next box writes. One last habit worth forming now: pip install rich and pip install rich==13.7.0 differ only by six characters, and only one of them will still mean the same thing in a year.pip install some-package can run arbitrary code on your machine at install time — before you've imported a thing. A pure-python wheel cannot; it is only unzipped. This is the security reason to prefer wheels and to install only from sources you trust, and it is the seed of the hash-verification you'll bolt on in section 6.rich-13.7.0-py3-none-any.whl to .zip and any archiver opens it. The .whl extension is a convention for pip, exactly as .py was a convention in Chapter 1: the machine reads the bytes and the layout, not the suffix.Unzipping a wheel is easy. The hard part came one step earlier: which version of each package should pip download? Ask for two libraries and pip must pick one version of each so that everyone's Requires-Dist pins agree at once. That is not a lookup. It is a search — and sometimes it hangs. →
05Dependency resolution is a constraint problem
You run pip install flask some-analytics-lib and pip stalls, printing "This is taking longer than usual... backtracking", sometimes ending in a wall of text titled ResolutionImpossible. It feels like it should just grab the latest of each and be done. Why is choosing version numbers genuinely hard — and why does "no solution exists" happen at all, instead of a clean error? Because resolution is search, not lookup, and this section makes you see the search.
Every release declares its own requirements as version constraints, in the Requires-Dist lines you saw in METADATA:
Requires-Dist: urllib3>=1.21.1,<3
Requires-Dist: certifi>=2017.4.17
Requires-Dist: charset-normalizer>=2,<4
Requires-Dist: idna>=2.5,<4Your top-level requirements, plus every transitive dependency's requirements, form a system of constraints over version variables. There is one variable per package, each with an allowed set of versions. And site-packages holds exactly one version per name — section 1's root cause, returning to collect its debt. So the resolver must find a single assignment: pick one version for every package such that every constraint on it holds simultaneously. That is a constraint-satisfaction problem, a SAT problem in disguise.
Put numbers on the word search. Suppose your install pulls in 20 packages, and each has 10 plausible versions. The space of assignments is 10 multiplied by itself 20 times, which is 10^20, a hundred billion billion combinations. The resolver plainly cannot test them all, so it prunes: every constraint it propagates kills whole branches at once. When the pruning bites you get an answer in a blink. When the constraints interlock badly it crawls, and that crawl is the backtracking message on your screen.
requests quietly pulls 5, tensorflow ~75, apache-airflow ~180 — all of it code you now run and trust.six up to apache-airflow. The number that matters is the gap between what you typed (always 1) and what actually landed. Each extra package is code from a maintainer you never chose, executed the instant you import it — and that swarm of packages, each with its own version constraints, is the search space section 5's resolver has to satisfy.pip's resolver (a library called resolvelib) does backtracking search. Choose a candidate version for a package. Propagate its constraints to narrow the allowed sets of the others. If any package's allowed set collapses to empty, backtrack: undo the most recent choice, drop to the next-older version, and try again. Repeat until a consistent assignment is found, or the search space is exhausted.
def resolve(work, chosen, allowed):
if not work:
return chosen # every package assigned -> success
pkg, rest = work[0], work[1:]
for ver in newest_first(allowed[pkg]):
narrowed = apply_deps(index[pkg][ver], allowed)
if narrowed is None: # some set went empty -> dead end
continue # BACKTRACK: try the next-older version
got = resolve(rest, {**chosen, pkg: ver}, narrowed)
if got: return got
return None # no candidate works -> ResolutionImpossibleWorst case, that search is exponential, which is exactly why pip can hang. It is not stuck but exploring a combinatorial tree of version choices. Historically the pain was worse: to read a version's Requires-Dist, pip sometimes had to download that version's wheel or sdist first, so every probe carried heavy network I/O. And ResolutionImpossible is not a crash but a proof. It means the constraint set is unsatisfiable: package A needs C<2 while package B needs C>=2. No single version of C can satisfy both, and no amount of backtracking will conjure one.
The classic pinch point is the diamond: your project depends on A and B, and both depend on a shared C. If A wants C>=1,<2 and B wants C>=1,<3, the overlap [1,2) holds a solution. Now tighten B to C>=2 and the overlap vanishes. The diamond becomes unsatisfiable, and pip hands you the proof.
Say the resolver wins and prints a clean set. There's a catch that section 6 exists to close: it re-solves every time, against an index that keeps growing new releases. Same requirements file, different day, different answer. To freeze the exact set forever — the same bytes on every machine — you need a lockfile. →
06Reproducibility: from requirements to a lockfile
"Works on my machine." Your requirements.txt says one word: flask. Six months later a teammate runs the same file on a fresh laptop, the resolver picks flask 3.1 instead of your 2.3, a transitive dependency crosses a major version, and CI turns red. Identical file, different bytes. The install has to become byte-for-byte identical everywhere and every time — and you deserve to know precisely which artifact delivers that guarantee. It is a ladder, and we climb it rung by rung.
Rung 1 — a loose name. flask means "whatever the resolver picks at install time," which is non-deterministic across time. PyPI keeps gaining new releases, and section 5's resolver re-solves against that moving index on every install. Nothing is pinned, so the ground moves under you.
Rung 2 — pin the direct dependency. flask==2.3.3 freezes your top level, but flask's own dependencies still float: jinja2, werkzeug, click, markupsafe. A werkzeug point release with a behavior change still slips in, so you've nailed the trunk and left the branches loose.
Rung 3 — a fully-resolved lockfile. pip freeze, or pip-tools' compiled requirements.txt, or uv.lock / poetry.lock, pins every package in the resolved graph to an exact version. This does something profound: it bypasses the resolver entirely at install time. The install stops being a search. It degrades into a straight download-and-unpack of a known set. Same graph, every machine, every day — because there is no longer any choice left to make.
freeze writes the file, a second machine replays it, and the two shelves match# ---------- write down what is on the shelf ---------- (.venv) > python -m pip freeze (.venv) > python -m pip freeze > requirements.txt (.venv) > type requirements.txt REM cat requirements.txt on POSIX # ---------- replay it into the SAME environment: a no-op, on purpose ---------- (.venv) > python -m pip install -r requirements.txt # ---------- replay it into a brand-new one. this is the whole point. ---------- (.venv) > deactivate > mkdir C:\tmp\fresh-clone > copy requirements.txt C:\tmp\fresh-clone\ > cd C:\tmp\fresh-clone > python -m venv .venv > .venv\Scripts\activate.bat (.venv) > python -m pip install -r requirements.txt (.venv) > python -m pip freeze
(.venv) > python -m pip freeze markdown-it-py==4.2.0 mdurl==0.1.2 Pygments==2.20.0 rich==13.7.0 <- every package, pinned. even the ones you never named. (.venv) > type requirements.txt markdown-it-py==4.2.0 mdurl==0.1.2 Pygments==2.20.0 rich==13.7.0 (.venv) > python -m pip install -r requirements.txt Requirement already satisfied: markdown-it-py==4.2.0 in c:\tmp\jukebox-project\.venv\lib\site-packages (from -r requirements.txt (line 1)) (4.2.0) Requirement already satisfied: mdurl==0.1.2 in c:\tmp\jukebox-project\.venv\lib\site-packages (from -r requirements.txt (line 2)) (0.1.2) Requirement already satisfied: Pygments==2.20.0 in c:\tmp\jukebox-project\.venv\lib\site-packages (from -r requirements.txt (line 3)) (2.20.0) Requirement already satisfied: rich==13.7.0 in c:\tmp\jukebox-project\.venv\lib\site-packages (from -r requirements.txt (line 4)) (13.7.0) (.venv) > python -m pip install -r requirements.txt # in the FRESH clone Collecting markdown-it-py==4.2.0 (from -r requirements.txt (line 1)) Using cached markdown_it_py-4.2.0-py3-none-any.whl.metadata (7.4 kB) Collecting mdurl==0.1.2 (from -r requirements.txt (line 2)) Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB) Collecting Pygments==2.20.0 (from -r requirements.txt (line 3)) Using cached pygments-2.20.0-py3-none-any.whl.metadata (2.5 kB) Collecting rich==13.7.0 (from -r requirements.txt (line 4)) Using cached rich-13.7.0-py3-none-any.whl.metadata (18 kB) Using cached markdown_it_py-4.2.0-py3-none-any.whl (91 kB) Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB) Using cached pygments-2.20.0-py3-none-any.whl (1.2 MB) Using cached rich-13.7.0-py3-none-any.whl (240 kB) Installing collected packages: Pygments, mdurl, markdown-it-py, rich Successfully installed Pygments-2.20.0 markdown-it-py-4.2.0 mdurl-0.1.2 rich-13.7.0 (.venv) > python -m pip freeze markdown-it-py==4.2.0 mdurl==0.1.2 Pygments==2.20.0 rich==13.7.0 <- identical to the file we started from. the loop closed.
freeze is character-for-character the file the first freeze wrote. That equality is the entire promise of a requirements file, and you just proved it rather than took it on faith. Look at what freeze chose to write. Four lines, all with ==, including the three packages you never typed. freeze does not record what you wanted; it photographs what is on the shelf, transitive dependencies and all, which is why it produces a graph a second machine can rebuild exactly. Now read the two install -r runs against each other, because they are the same command doing different work. In the original venv every line came back Requirement already satisfied — pip compared the pins to what was installed, found them equal, and wrote nothing. In the fresh clone the same file produced Collecting for all four and then one Installing collected packages. That is what Section 6 meant by the install degrading into a download: with every version pinned, there is no search left to run, so the resolver has nothing to decide. Notice what this file is not, though, and be honest about it. There are no hashes here, so nothing verifies the bytes, and there is no record of which line you asked for versus which line rich dragged in. Delete rich from the file next month and you are left hand-editing three orphans you never chose. That gap is exactly what pyproject.toml and a compiled lockfile close, and it is why the professionals keep two files instead of one. For now, form the habit: freeze after every deliberate install, commit the file, and never hand-write a version into it.Rung 4 — content hashes. Add --hash=sha256:... to each line. Before installing, pip computes the sha256 of the downloaded artifact and refuses to proceed unless that matches the recorded hash. The install becomes tamper-evident: a swapped PyPI file, a poisoned mirror, a man-in-the-middle — all rejected at the gate.
flask==2.3.3 \
--hash=sha256:09c347a92aa7ff4a8e7f3206795f30d826654baf38b873d0744cd571ca609efc
werkzeug==2.3.7 \
--hash=sha256:2b8c0e447b4b9dbcc85dd97b6eeb4dcbaf6c8b6c00f7bf1932d2b1b0b0b0b0b0
jinja2==3.1.2 \
--hash=sha256:6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61
# every package pinned; every artifact verified before it landsNotice the layering. Pinned versions give you the same graph, the lockfile gives you the same files, and the hashes give you the same, verified, bytes — three independent guarantees, stacked. And this is where the chapter's law finally shows its full face: what crosses the boundary into your environment is always bytes. Reproducibility is nothing but the discipline of proving those bytes are the ones you meant.
# ============================================================
# fresh_env_session -- the whole chapter, start to finish, once.
# the jukebox from Chapter 21 is about to grow a dependency.
# ============================================================
# 0. what does THIS machine already have? (there is a rich here already)
> python -c "import importlib.metadata as m; print(m.version('rich'))"
# 1. make the project its own shelf
> python -m venv .venv
# 2. step onto it
> .venv\Scripts\activate.bat REM cmd.exe
$ source .venv/bin/activate # macOS / Linux
# 3. install ONE pinned dependency, with the interpreter's own pip
(.venv) > python -m pip install rich==13.7.0
# 4. ask the same question again -- from inside
(.venv) > python -c "import importlib.metadata as m; print(m.version('rich'))"
# 5. write the shelf down so anyone can rebuild it
(.venv) > python -m pip freeze > requirements.txt
(.venv) > type requirements.txt
# 6. step off
(.venv) > deactivate
# 7. and ask one final time -- from outside
> python -c "import importlib.metadata as m; print(m.version('rich'))\"
> python -c "import importlib.metadata as m; print(m.version('rich'))"
13.7.1 <- the base install's rich
> python -m venv .venv
> .venv\Scripts\activate.bat
(.venv) > python -m pip install rich==13.7.0
Collecting rich==13.7.0
Using cached rich-13.7.0-py3-none-any.whl.metadata (18 kB)
Collecting markdown-it-py>=2.2.0 (from rich==13.7.0)
Collecting pygments<3.0.0,>=2.13.0 (from rich==13.7.0)
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich==13.7.0)
Installing collected packages: pygments, mdurl, markdown-it-py, rich
Successfully installed markdown-it-py-4.2.0 mdurl-0.1.2 pygments-2.20.0 rich-13.7.0
(.venv) > python -c "import importlib.metadata as m; print(m.version('rich'))"
13.7.0 <- the PROJECT's rich
(.venv) > python -m pip freeze > requirements.txt
(.venv) > type requirements.txt
markdown-it-py==4.2.0
mdurl==0.1.2
Pygments==2.20.0
rich==13.7.0
(.venv) > deactivate
> python -c "import importlib.metadata as m; print(m.version('rich'))"
13.7.1 <- untouched. it never knew.
13.7.1 -> 13.7.0 -> 13.7.1
one machine, one name, two versions, zero conflict
site-packages has exactly one slot per import name, so two versions of rich could never coexist. They are coexisting here, three lines apart, and nothing was overwritten. The constraint was never wrong; we simply stopped sharing the store. Walk the steps and notice how little machinery that took. Step 1 wrote a folder. Step 2 reordered a string. Step 3 unzipped four wheels into a directory that belongs to this project alone. Step 6 put the string back. Every one of those is something you could do by hand with a file manager, and none of them is a service, a daemon, or a registry. The isolation you get is total, and its total cost is a directory and a three-line text file. Step 7 is the one worth staring at longest. The base interpreter reports 13.7.1 after all of it, exactly as it did before, because nothing we ran ever wrote outside .venv. That is the actual meaning of "do not sudo pip install" — not a superstition about privileges, but a claim about which directory your changes are allowed to reach. And step 5 leaves the receipt. requirements.txt is four pinned lines that let a stranger rebuild this exact shelf, on a different operating system, next year. Two things to try. Re-run step 3 without the pin and watch a newer rich land, then diff the new freeze against the old one. Then delete the whole .venv folder, remake it, replay requirements.txt, and confirm you are back where you started — the environment is disposable, and the file is the thing you keep.pyproject.toml — the file that gives your folder a nameChapter 21 built the package; this is the one file that lets pip install it, and where your dependencies are declaredpyproject.tomlOne file, one standard (PEP 621), read by pip, build backends, linters and test runners. import never opens it.[project]Your package's identity: name, version, and the dependency list. This is where your Requires-Dist is born.dependenciesThe abstract requirement — what you want. It is the input to Section 5's resolver, not its output.[build-system]Which backend turns the folder into a wheel. pip installs it in an isolated environment first, then calls it.name = "jukebox" is what pip installs; import jukebox finds the folder. They may legally differ, and often do.-e (editable)Installs a pointer, not a copy. Edit tracks.py and the next import sees it — no reinstall, no stale duplicate.optional-dependencies, entry points and version schemes. All real, all later — the file above already works.you type
# ---------- pyproject.toml, at the repo root ----------
[project]
name = "jukebox"
version = "0.1.0"
description = "The playlist package from Chapter 21."
requires-python = ">=3.9"
dependencies = ["rich==13.7.0"]
[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"
# ---------- install the project into its own venv ----------
(.venv) > python -m pip install -e .
(.venv) > python -m pip show jukebox
(.venv) > python -m pip list
# ---------- then import it from ANY directory on the disk ----------
(.venv) > cd C:\tmp\anywhere
(.venv) > python -c "import jukebox; print(jukebox.as_clock(245))\"you see
(.venv) > python -m pip install -e .
Obtaining file:///C:/tmp/jukebox-project
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Requirement already satisfied: rich==13.7.0 in c:\tmp\jukebox-project\.venv\lib\site-packages (from jukebox==0.1.0) (13.7.0)
Building wheels for collected packages: jukebox
Building editable for jukebox (pyproject.toml): finished with status 'done'
Created wheel for jukebox: filename=jukebox-0.1.0-0.editable-py3-none-any.whl size=2691
Successfully built jukebox
Installing collected packages: jukebox
Successfully installed jukebox-0.1.0
(.venv) > python -m pip show jukebox
Name: jukebox
Version: 0.1.0
Summary: The playlist package from Chapter 21.
Location: C:\tmp\jukebox-project\.venv\Lib\site-packages
Editable project location: C:\tmp\jukebox-project
Requires: rich <- read straight off [project].dependencies
Required-by:
(.venv) > python -m pip list
Package Version Editable project location
-------------- ------- -------------------------
jukebox 0.1.0 C:\tmp\jukebox-project
markdown-it-py 4.2.0
mdurl 0.1.2
pip 24.2
Pygments 2.20.0
rich 13.7.0
(.venv) > cd C:\tmp\anywhere
(.venv) > python -c "import jukebox; print(jukebox.as_clock(245))"
4:05 <- no sys.path surgery. it is installed.pyproject.tomlis invisible toimport. It changes what pip knows about your project, and nothing aboutsys.pathat runtime.- The dot in
pip install -e .is a path, not a flag. It means "the project in this directory". - Without
[build-system], pip guesses a legacysetuptoolsbackend. Write the two lines and stop depending on a fallback. dependenciesholds abstract requirements;requirements.txtholds a resolved snapshot. Keep both — they answer different questions.pip freezeon an editable install emits-e c:\tmp\jukebox-project, a machine-local path. Never ship that line to a teammate.requires-pythonconstrains the interpreter, and pip enforces it before downloading. It is not a dependency, and it has no version to install.
The professional discipline is two files, not one. An abstract requirements.in holds what you want — flask, loosely. You compile it once, with the resolver, into a locked requirements.txt. That file spells out exactly what you get: the full pinned, hashed closure. You regenerate the lockfile only when you deliberately choose to upgrade, never as a silent side effect of an install. uv and Poetry bake this split into their *.lock files, and pip-tools does it with pip-compile. That is the exact moment "works on my machine" dies.
> reports\Scripts\python.exe app.py No Python at '"C:\Users\ajair\AppData\Local\Programs\Python\Python312\python.exe'
reports\Scripts\python.exe is a launcher, it opened pyvenv.cfg, read the home key, and went to collect the real interpreter from that address. Nothing is there, so there is nothing to launch, and the process exits with status 103 before one line of app.py is read. What earns this its own row is what it proves. A venv holding its own Python could not fail this way — it would simply run. This one cannot start at all, because the only Python it ever had was a path written in a text file. (To stage it on one machine we pointed home at a directory that is not on this disk. In the wild the same line arrives three ways: you upgraded or uninstalled the base Python, you moved the install, or somebody committed .venv to git and you cloned it onto a laptop where Python lives elsewhere.) The pip.exe beside it dies identically, with the same message and the same 103, which tells you the shims are launchers too. Two small things worth noticing. The stray " after the opening quote is not a typo in your path — it is the launcher showing you the command line it was assembling. And on macOS and Linux bin/python is usually a symlink rather than a launcher, so the identical break arrives in the shell's voice instead, refusing a file that is not there.python -m venv .venv, then python -m pip install -r requirements.txt; and never commit .venv, because the file that makes it work is a set of absolute paths belonging to one disk> python -m pip install "requests==2.20.0" "urllib3==2.0.0"
INFO: pip is looking at multiple versions of requests to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install requests==2.20.0 and urllib3==2.0.0 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested urllib3==2.0.0
requests 2.20.0 depends on urllib3<1.25 and >=1.21.1
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflictsThe user requested urllib3==2.0.0 is a set with exactly one member. requests 2.20.0 depends on urllib3<1.25 and >=1.21.1 is the interval [1.21.1, 1.25), read straight off that release's METADATA. Intersect a single point at 2.0.0 with an interval that stops below 1.25 and you get the empty set — which is section 1's opening story about these two exact packages, now in pip's own handwriting rather than ours. Then read the INFO line above it, because that line is the backtracking you stepped through. You pinned urllib3 and left requests free to move, so the resolver walked back through requests releases hoping one of them carried a pin that admits 2.0.0, and ran out of candidates. Three details repay a second look. The report names one pair, not every constraint in the graph, so a second conflict can be queued behind the one you just fixed. The two numbered suggestions are ordered by how much they cost you: loosening a range you wrote yourself is honest, while removing a pin hands the choice back to the resolver and to whatever PyPI holds next month. And notice which of the two constraints you are even able to edit — urllib3==2.0.0 is yours; urllib3<1.25 is inside a wheel published years ago, and no flag on your command line can reach it.--dry-run before you commit to it — here that means dropping urllib3 from the command entirely and letting requests' own range pick the version> python -m pip install -r requirements.txt
Collecting mdurl==0.1.2 (from -r requirements.txt (line 1))
Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
mdurl==0.1.2 from https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl (from -r requirements.txt (line 1)):
Expected sha256 6a8f6804087b7128040b2fb2ebe242bdc2affaeaa034d5fc9feeed30b443651b
Got 84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB), tells you the bytes already reached your disk; pip then hashed them and compared. Expected is the string sitting in your requirements file. Got is the sha256 of what actually arrived. They differ, so the artifact is refused — after the download and before site-packages. That is worth holding onto, because it locates the gate exactly: it stands between the network and the shelf, which is why a rejected install leaves nothing to clean up. pip then offers its two readings in order of likelihood, and the first is nearly always the true one: you bumped a version and left the old hash sitting underneath it. That is literally how this capture was made — we pinned 0.1.2 above 0.1.1's recorded hash, which is the same slip as editing a version by hand in a file you did not recompile. The second reading is the one the whole mechanism exists for. If you changed nothing, then the bytes at that URL are not the bytes that were there when the lock was written, and retrying cannot help, because sha256 does not negotiate. Finally, notice how much the message hands you: the exact URL it pulled, so you can fetch and hash it yourself, and (from -r requirements.txt (line 1)), so you know which line is lying.pip-compile --generate-hashes or uv lock rewrites the version and its hash as one act, and a lockfile you hand-edit is a lockfile you have already broken== or >= is the honest answerjukebox/, pyproject.toml, and a requirements.txt reading markdown-it-py==4.2.0, mdurl==0.1.2, Pygments==2.20.0, rich==13.7.0. There is no .venv — they were right not to commit one. Write the exact commands, in order, for Windows and for macOS or Linux, and then a third form that uses no activation at all, the way a Dockerfile would. Then answer three questions in writing before you run anything. Would running pip install -r requirements.txt as your first command work, and what would it actually do? Is that four-line file sufficient to reproduce their environment, and name three things it does not record. And what one command proves your rebuild succeeded, byte for byte? Second, pin or float — three scenarios, and they do not share an answer. A. A web service you deploy to a server every Friday. B. A library you publish to PyPI for strangers to pip install alongside their own dependencies. C. A data-analysis notebook you will reopen in six months to regenerate one figure. For each, write the dependency lines you would actually ship — == or >= or a bounded range — and say which file they belong in. Third, and this is the part that matters, justify each one from the mechanism. Not from a style guide. Section 1 proved there is one physical slot per import name; Section 5 proved the resolver must satisfy every constraint at once. Use those two facts to explain why exactly one of A, B and C must not be pinned, and what specific failure your pin would cause for someone you will never meet. One hint and no more: ask, for each scenario, whether anything else will ever be installed beside your code in the same site-packages.show the solution
# =====================================================================
# PART 1 -- recreate the environment from requirements.txt alone
# =====================================================================
# ---------- Windows, cmd.exe ----------
> cd C:\code\jukebox-project
> python -m venv .venv
> .venv\Scripts\activate.bat
(.venv) > python -m pip install -r requirements.txt
(.venv) > python -m pip freeze # verify: must match the file, line for line
# ---------- macOS / Linux ----------
$ cd ~/code/jukebox-project
$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ python -m pip install -r requirements.txt
(.venv) $ python -m pip freeze
# ---------- no activation at all (cron, CI, a Dockerfile) ----------
> python -m venv .venv
> .venv\Scripts\python.exe -m pip install -r requirements.txt
$ .venv/bin/python -m pip install -r requirements.txt
# The three answers.
#
# 1. NO -- do not run `pip install -r requirements.txt` first. There is no venv
# yet, so `pip` is whatever PATH finds: probably the system interpreter. You
# would install four packages into a shelf you do not own. Order is load-bearing:
# create, activate, THEN install.
#
# 2. The file is enough, and that is the whole point of it. It pins every package
# in the graph, so `install -r` never runs a search -- it downloads a known set.
# What the file does NOT carry: the Python version, the OS, any compiled-wheel
# platform tag, and any package installed by hand and never frozen.
#
# 3. `pip freeze` on the rebuilt env must be byte-identical to requirements.txt.
# If it is not, something was installed outside the file -- diff them and find out.
# =====================================================================
# PART 2 -- pin or float? three scenarios, three different right answers
# =====================================================================
# ---------- A. the deployed application ----------
requirements.txt:
rich==13.7.0
requests==2.32.3
urllib3==2.2.2
#
# PIN, hard, and every transitive package too.
# WHY: an application is deployed, not consumed. Nothing imports it, so no one
# else's constraints have to agree with yours -- you are free to be maximally
# specific. What you gain is that the thing you tested is the thing that runs.
# You upgrade by regenerating the file deliberately, on a day you chose, with
# tests in front of you. A float here means production drifts while you sleep.
# ---------- B. the library you publish ----------
pyproject.toml:
dependencies = ["rich>=13,<14", "requests>=2.28"]
#
# FLOAT, inside honest bounds. Never `==` in a library.
# WHY: your library is INSTALLED ALONGSIDE other packages, into one site-packages
# with one slot per name (Section 1). `rich==13.7.0` forbids every other package
# in that environment from wanting any other rich. Two libraries pinning exactly
# is Section 5's ResolutionImpossible, manufactured by you.
# The bounds still say something true: >=13 is the oldest you tested,
# <14 is the next major, where the author is allowed to break you.
# ---------- C. the notebook you will open again in six months ----------
requirements.txt:
pandas==2.2.2
numpy==1.26.4
matplotlib==3.8.4
#
# PIN. This one surprises people, so be clear about why.
# WHY: the value of the artefact is that it still RUNS and still produces the
# same figure. There is no consumer to conflict with -- it is an application of
# one. Float it, and in six months numpy has crossed a major version, a pandas
# API you used is gone, and the notebook is archaeology. The pin is not caution;
# it is the only thing making the result reproducible.
# =====================================================================
# THE RULE, IN ONE LINE
# =====================================================================
# Pin what you DEPLOY. Bound what you PUBLISH.
# Applications and notebooks are deployed; libraries are published.
# `==` is a promise to yourself; `>=,<` is a promise you can keep to strangers.flaskrequestsabstract, human-edited, loose
flask==2.3.3 --hash=...+ 11 pinned transitive depscompiled, machine-generated, exact
requirements.txt is a wish; a compiled, hashed lockfile is a guarantee that the same bytes land in every site-packages on Earth. Isolation (the venv) gives each project its own shelf; the lockfile stamps identical contents onto every copy of that shelf.pip freeze already a lockfile? Almost, but it lies by omission. freeze dumps whatever is currently installed — with no hashes, and no way to tell your direct dependencies from the transitive ones they dragged in. A real lock (pip-tools, uv, Poetry) records both the hashes and the reason each package is present, so you can later change what you want without hand-editing what you got.You've closed the last leak. The venv gave each project a private shelf; pyvenv.cfg made it isolated; the wheel is bytes on disk; the resolver chose a consistent set; the lockfile froze those exact bytes onto every machine. The same law governed all of it — objects live inside one process, but only bytes cross a boundary. The next boundary this volume crosses is the one where your program stops being alone and starts running beside others — other cores, other processes — and the same flatten-and-rebuild machine will be waiting. →
- A
site-packagesis a key/value store with one physical slot per import name, andpip installis a mutating write into it — so on one shared shelf two projects with incompatible pins can never both win, and no amount of care changes that, because the data model has nowhere to put a second version. - A virtual environment is a directory of inert files, never a running thing: a launcher or symlink standing in for the interpreter, a three-line
pyvenv.cfg, and an emptysite-packages— andactivateis an ordinary shell script that prepends one directory to a string, which is exactly why spelling out.venv/bin/pythonbuys the identical isolation with no activation at all. - The isolation is computed in the interpreter's first milliseconds: finding
pyvenv.cfgpointssys.prefixat the venv while itshomekey keepssys.base_prefixon the base install, so the standard library is borrowed and the packages are private — andsys.prefix != sys.base_prefixis not a clever test for "am I in a venv", it is the definition. - A wheel is a ZIP with a five-field name, and those fields tell you before anything runs whether an install is an unzip (
py3-none-any) or a build (cp312-...-manylinux) — the one-second install and the one-minute compile are the same act, differing only in whether the wheel already existed, and theRECORDwritten on the way in is what makes uninstall exact instead of a guess. - Choosing versions is backtracking search, not lookup, which is why pip can crawl and why
ResolutionImpossibleis a proof rather than a crash — and a lockfile ends the search by pinning every package in the resolved graph, with--hashgoing one step further and fixing the bytes, not merely the numbers.
.py is a convention and the machine reads the layout rather than the suffix, which is the whole reason renaming a .whl to .zip opens it in any archiver; chapter 3 gave you binding, so "one slot per name" lands as a fact about names you already knew rather than a rule pip invented; chapter 7 gave you the hash table, and site-packages turns out to be one, keyed by import name — while the sha256 on a locked line borrows the same idea for a different job, and the difference is worth saying out loud, since a dict's hash may collide cheaply and a lock's hash must not; chapter 13 taught you to read a traceback as information, which is what turns the resolver's wall of text into the two lines that actually fight; chapter 17 separated bytes from text, and chapter 18 gave you the law this chapter never stopped obeying — objects live inside one process, and only bytes cross a boundary; chapter 20 gave you sys.path as an ordered list resolved first-hit, without which a venv is simply inexplicable; and chapter 21 gave you the package itself, the thing you can now name, pin, and hand to a stranger. Chapter 22 added no new law. It added a shelf per project and a receipt for what is on it — so that "it works on my machine" stops being a defence and becomes a claim somebody else can check.Where does `import requests` actually find requests, and why does the same code run clean on one machine and break on another? This chapter makes environments concrete: a package name is a lookup that only one version can win, a venv is a folder with a text file, a wheel is a zip with a receipt, and reproducibility is a hash you either match or you don't. Every widget below is a small, runnable model of the real machinery.