17Bytes vs text — Unicode and UTF-8 at the byte level
In Chapter 16 we made a write survive — flush, fsync, the page cache, all the machinery it takes to be sure the bytes really reach the disk. But we never stopped to ask what those bytes were. In this one we do, and I want to take it slowly. There's a wall hiding in here that Python 3 put up on purpose: the wall between text and bytes. Most tutorials climb straight over it without ever naming it. Here's the plan. We'll set a str and a bytes object side by side and watch four visible letters turn into five integers. Then we'll derive UTF-8 by hand, one bit at a time, until there's no black box left in it. And the whole way through we keep asking the one thing that actually matters — what really crosses the line the moment a string leaves your process? The answer never changes. What crosses a process boundary is always bytes, never objects — a flat run of numbers from 0 to 255. Nothing is left of the rich, structured str that RAM was holding. Get that crossing wrong and you get mojibake: not corruption, not a Python bug, just a decoder reading perfectly good bytes against the wrong table. By the end you'll have derived the whole of UTF-8 with your own hands, and never be surprised by an encoding again.
01Two alphabets: code points vs 0–255
Let's start with the smallest thing that cracks the illusion open. Open a fresh prompt and type this: four letters you can see. Then watch what comes back:
>>> "café".encode("utf-8")
b'caf\xc3\xa9' # four letters → five bytes, one shown as \xNN
>>> len("café"), len("café".encode("utf-8"))
(4, 5)Look at what it did. You've spent your whole programming life treating text and a file's contents as the same substance — you read a file, you get text; you print text, it lands in a file. Python 3 flatly refuses to let those two blur. The é you see as one letter came back as two bytes, \xc3\xa9 — and the object holding them is not a string at all. It's a different kind of thing that merely prints in a way that fools you. That's the wall this whole chapter is about, and until it's concrete in your hands every codec idea here has nowhere to stand. So let's make it concrete.
A Python str is a C struct called PyUnicodeObject. Internally it holds an array of code points — abstract character-identity numbers — plus a one-word kind tag. That tag is PEP 393's flexible string representation. It lets the array store each code point in the narrowest slot that fits the widest character present. So you get 1 byte per character if every code point fits in Latin-1 (kind 1). You get 2 bytes if the largest fits in the Basic Multilingual Plane (kind 2, UCS-2). You get 4 bytes otherwise (kind 4, UCS-4). Here's the crucial fact. When you write s[i] you get back a length-1 str, and ord(s[i]) is a code point somewhere in the range 0 to 0x10FFFF — about 1.1 million possible values. The str alphabet is over a million symbols wide.
A bytes object is a completely unrelated C struct, PyBytesObject: a flat C char array of raw uint8 values. When you write b[i] you get back a plain Python int, and that int is somewhere in 0 to 255 — exactly 256 possible values, no more. The bytes alphabet is 256 symbols wide, full stop. So a str and a bytes are two distinct types over two distinct alphabets: ~1.1 million code points versus exactly 256 byte values. They are as different as a musical score is from the grooves cut in a vinyl record. One is meaning; the other is a physical encoding of it.
.encode / .decode are the only doors between the two alphabets, and mixing them directly raises TypeError.Python 2 blurred this wall and paid for it. There, 'x' + u'y' silently decoded the byte string using ASCII to promote it to Unicode. That worked on your English test data. Then it detonated with a UnicodeDecodeError in production the first time a real name arrived. Python 3 tore that bridge down deliberately. Now 'x' + b'y' raises a TypeError immediately — no default codec, no ASCII guess, no runtime surprise. There is no implicit conversion in either direction, ever. The only two functions that cross the wall are str.encode and bytes.decode, and each one demands that you name a codec. That refusal is the feature.
"caf" + b"\xc3\xa9" raises TypeError on the spot: no ASCII guess, no silent promotion, no waiting until production to find out. That refusal is the feature, you just said so. Then if I ever hand a decode the wrong codec, it will refuse me there too — a decode either gives me back my text or it blows up in my face. My importer ran clean this morning, so the names it loaded are right.>>> b = "café".encode("utf-8") # correct bytes, from a correct program
>>> "caf" + b"" # EMPTY bytes: the wall reads no content at all
>>> b.decode("latin-1") # the WRONG key. Same promise?
>>> [sum(bytes([i]).decode(c, "replace").count("�") for i in range(256))
... for c in ("latin-1", "cp1252", "utf-8")]Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can only concatenate str (not "bytes") to str 'café' [0, 5, 128]
"caf" + b"" it never read a single byte — and line two proves that, because the bytes object sitting there is empty. There was no content to inspect. It compared two type pointers, saw str on one side and bytes on the other, and stopped. That check is free, and it is total. For b.decode("latin-1") it did exactly what you asked it to do: mapped byte 0xc3 to U+00C3, mapped byte 0xa9 to U+00A9, and handed you back a five-character string that is, on its own terms, perfectly well-formed. And it could not have known better, because there is nothing inside C3 A9 that says utf-8. A byte stream carries no record of the table it was written against — that is what makes it a byte stream — so "did I name the right codec" is a question no codec is ever asked. What a strict decode checks is grammar, and grammar is a property of the codec you named, not of whether you named the right one. Line four measures how little that grammar buys you. The leading 0 is no surprise, and section 04 will say it plainly: latin-1 has a meaning for all 256 bytes, so it cannot refuse anything. The number worth carrying is the spread beside it — cp1252 refuses exactly five of the 256 (81 8D 8F 90 9D), and UTF-8 refuses 128 of them as lone bytes. How loudly a wrong key fails is a property of which key you named, and it runs from never to half the time. Two labelled refinements, so this is not a half-truth. First, a wrong key is not always silent: point the same mistake the other way — cp1252 bytes read as UTF-8 — and it usually crashes at a named byte offset instead of garbling, which is the asymmetry section 05 is built on. Not always, though. Cp1252 bytes that happen to spell valid UTF-8 slip through in silence, and that is precisely why a mojibake string can be repaired at all — a repair you will write by hand in section 05. So a wrong key is sometimes loud. It is never reliably loud, and you cannot build a habit on sometimes. Second, the exception was never the guarantee — the key is. What Python genuinely promises is the round-trip law in the very next card, s.encode(c).decode(c) == s, for the same c — a promise about a name you write down, not a check the interpreter can run for you afterwards. That is why "name the codec at every crossing" is not fussiness or ceremony. It is the only place in the whole system where the guarantee can live.The thing that fools everyone is the repr. When Python shows you b'caf\xc3\xa9', that leading b is a signpost: this is a sequence of integers. It's rendered as ASCII where a byte happens to be a printable ASCII character, and as \xNN where it isn't. It is a display convenience, not text. list(b'caf\xc3\xa9') unmasks it as [99, 97, 102, 195, 169] — five integers, nothing more. The c, a, f you see are just bytes 99, 97, 102 that happen to fall in the printable ASCII range. The machine has no idea they "spell" anything.
bytes. Nothing is shared with s; you now hold two objects.\xNN escape, which is the language telling you these are numbers.'utf-8' in Python 3. Type it anyway — the habit is what survives being copied into a script that runs somewhere else.s.encode(c).decode(c) == s for any codec that covers s. Same key both sides, and you get your string back exactly.you type
# doors.py -- the only two functions that cross the wall, and what they cost
s = "Zoë Müller — 30€" # str: code points you can SEE
b = s.encode("utf-8") # door 1: str -> bytes
print("s :", type(s).__name__, "|", s)
print("b :", type(b).__name__, "|", b)
print("len(s) :", len(s), "code points")
print("len(b) :", len(b), "bytes")
print("b.hex() :", b.hex(" "))
back = b.decode("utf-8") # door 2: bytes -> str
print("round-trip:", back == s, "| same key both sides")
lit = b"Zo\xc3\xab" # the b'' literal: bytes you type by hand
print("literal :", lit, "==", "Zoë".encode("utf-8"), "->", lit == "Zoë".encode("utf-8"))
for codec in ("utf-8", "utf-16", "latin-1"):
try:
print(f"{codec:<8} : {len(s.encode(codec)):>2} bytes")
except UnicodeEncodeError as e:
print(f"{codec:<8} : UnicodeEncodeError - cannot encode {e.object[e.start:e.end]!r}")
try:
s + b
except TypeError as e:
print("s + b :", type(e).__name__ + ":", e)you see
s : str | Zoë Müller — 30€
b : bytes | b'Zo\xc3\xab M\xc3\xbcller \xe2\x80\x94 30\xe2\x82\xac'
len(s) : 16 code points
len(b) : 22 bytes
b.hex() : 5a 6f c3 ab 20 4d c3 bc 6c 6c 65 72 20 e2 80 94 20 33 30 e2 82 ac
round-trip: True | same key both sides
literal : b'Zo\xc3\xab' == b'Zo\xc3\xab' -> True
utf-8 : 22 bytes
utf-16 : 34 bytes
latin-1 : UnicodeEncodeError - cannot encode '—'
s + b : TypeError: can only concatenate str (not "bytes") to str
b"café"is aSyntaxError— bytes literals are ASCII-only. Writeb"caf\xc3\xa9", or encode the str.bytes("abc")raisesTypeError: string argument without an encoding. The constructor refuses to guess, exactly like the doors do.- Only
strhas.encode; onlybyteshas.decode. There is nos.decodein Python 3, and that is deliberate. str(b)quietly gives you the repr, the text"b'Zo\xc3\xab'". It never errors and it is almost always a bug;python -bwarns about it.- Latin-1 cannot encode
—or€at all, so that failure lands on the encode side asUnicodeEncodeError, not on decode. bytes(4)is four zero bytes, not the number four.bytes([4])is the one byteb'\x04'.
str is code points (an alphabet of ~1.1 million); bytes is integers 0–255 (an alphabet of 256). They are different types over different alphabets, and Python 3 will never convert one to the other behind your back. The only doors are .encode(codec) and .decode(codec) — and each one makes you name the key. Every other idea in this chapter is a detail about those two doors.b[i] is an int, then iterating a bytes gives you integers, but slicing it gives you back bytes. So b'caf\xc3\xa9'[4] is 169 (an int) while b'caf\xc3\xa9'[4:5] is b'\xa9' (a one-byte bytes). Same object, two different return types depending on whether you index or slice — a tiny asymmetry that trips up everyone parsing binary formats.✗ The myth
"A bytes object is just a string of characters with a b in front — same content, different label."
✓ The reality
They are unrelated C structs. str stores code points up to 0x10FFFF; bytes stores integers up to 255. The pretty b'caf\xc3\xa9' is really [99,97,102,195,169]. Nothing about it "is" the letters — until you decode it with the right key.
You now have two clearly separated shores. Next: what actually lives on the left shore — a code point isn't a letter and it isn't a byte, it's a third thing the Unicode standard invented to end the chaos. →
b[3:4] is b'\xab'; b[3] is 171. Same object, two answers.hex(" ") groups by byte for reading; fromhex ignores whitespace so you can paste a dump straight back.b[0] = 0x59 raises TypeError. Like str, every "edit" builds a new object, which is what makes bytes safe to share and to hash.append, extend and slice assignment. Build a packet byte by byte in one, then freeze it with bytes(ba).b'Zo\xc3\xab' prints printable ASCII as letters and everything else as \xNN. It is a display convenience, not evidence of text.you type
# bytesobj.py -- the bytes type, poked from every side
b = "Zoë".encode("utf-8") # b'Zo\xc3\xab'
print("b :", b)
print("b[3] :", b[3], "|", type(b[3]).__name__, " <- INDEX gives an int")
print("b[2:4] :", b[2:4], "|", type(b[2:4]).__name__, "<- SLICE gives bytes")
print("list(b) :", list(b))
print("len(b) :", len(b))
print("b.hex() :", b.hex())
print("b.hex(' ') :", b.hex(" "))
print("fromhex :", bytes.fromhex("5a 6f c3 ab"))
print("b'Zo' in b :", b"Zo" in b)
print("b.find(b'\\xc3'):", b.find(b"\xc3"))
print("b.upper() :", b.upper(), " <- ASCII-only; c3 ab untouched")
try:
b[0] = 0x59
except TypeError as e:
print("b[0] = 0x59 :", type(e).__name__ + ":", e)
ba = bytearray(b) # the mutable twin
ba[0] = 0x59
ba.append(0x21)
print("bytearray :", ba)
print("back to str :", bytes(ba).decode("utf-8"))
print("mv[3] :", memoryview(b)[3], " <- zero-copy view, still ints")you see
b : b'Zo\xc3\xab'
b[3] : 171 | int <- INDEX gives an int
b[2:4] : b'\xc3\xab' | bytes <- SLICE gives bytes
list(b) : [90, 111, 195, 171]
len(b) : 4
b.hex() : 5a6fc3ab
b.hex(' ') : 5a 6f c3 ab
fromhex : b'Zo\xc3\xab'
b'Zo' in b : True
b.find(b'\xc3'): 2
b.upper() : b'ZO\xc3\xab' <- ASCII-only; c3 ab untouched
b[0] = 0x59 : TypeError: 'bytes' object does not support item assignment
bytearray : bytearray(b'Yo\xc3\xab!')
back to str : Yoë!
mv[3] : 171 <- zero-copy view, still ints
for x in byields integers, soif x == "Z"is alwaysFalse. Compare against90, or slice first and compare tob"Z".b.find("Zo")raisesTypeError. Bytes methods take bytes arguments; there is no automatic promotion of a str.b.upper()touches ASCII letters and nothing else. Bytesc3 absail through — there is no "case" in a number.bytearrayis unhashable, so it can never be a dict key. Freeze it withbytes()first.- A
bytesslice is a real copy; amemoryviewslice is not. Mutating the source under a live view changes what the view reports. - Concatenating bytes in a loop copies the whole buffer each time. Use a
bytearray, or collect a list andb"".joinit once.
02Unicode decouples character from storage
Here is the chaos the last section left you with. The character ë takes 1 byte in one file and 2 bytes in another. Two é's that look pixel-identical fail an == test. You paste a name from a web form and its length comes back "wrong." It feels like the machine is improvising. It isn't. There is one organising idea that dissolves all of it, and once it's installed the next three sections become mere storage trivia. The idea is this: every character has a single, fixed, storage-independent identity, assigned once by the Unicode standard — and everything else is a downstream choice.
Unicode, at its core, is a giant numbered lookup table. It defines a code space of exactly 1,114,112 slots — the code points U+0000 through U+10FFFF. Those are carved into 17 planes of 65,536 each. Plane 0, the Basic Multilingual Plane, holds almost every character of every living language. The astral planes 1 through 16 hold emoji, historic scripts, and math alphanumerics. A code point is an abstract integer identity permanently bound to a character NAME and a set of properties. U+00EB is, forever and everywhere, LATIN SMALL LETTER E WITH DIAERESIS. That binding says nothing whatsoever about bytes. It is pure identity — a museum accession number for the character.
Python hands you this exact map through three built-ins. ord('ë') goes character → code point and returns 235 (that's 0x00EB). chr(235) goes the other way, code point → character, returning 'ë'. And unicodedata.name('ë') returns the official standard name, 'LATIN SMALL LETTER E WITH DIAERESIS'. Try it:
import unicodedata
for ch in "ë€🌍": # e-diaeresis, euro sign, globe emoji
print(ch, hex(ord(ch)), unicodedata.name(ch))
# ë 0xeb LATIN SMALL LETTER E WITH DIAERESIS
# € 0x20ac EURO SIGN
# 🌍 0x1f30d EARTH GLOBE EUROPE-AFRICA ← above U+FFFF: an astral characterU+00EB has many byte encodings, and the same glyph can even be composed as base e plus a combining mark.str is already holding.chr(ord(c)) == c always holds, which makes the pair a lookup table you can trust.\u takes four hex digits and stops at U+FFFF. Capital \U takes eight and reaches the astral planes, where the emoji live."\N{EURO SIGN}" says what it means, and a typo is a SyntaxError at compile time rather than a wrong glyph in production.str it is code point 65; in a bytes it is the byte 65. Identical spelling, two alphabets — the wall again.name gives you the official title of a character; lookup turns that title back into the character.ord(ch) // 0x10000 tells you which of the seventeen planes a character lives in. Plane 0 is nearly every living script; plane 1 is where emoji sit.you type
# identity_ops.py -- the number under every character, both directions
import unicodedata
for ch in "Aë€🌍":
print(f"{ch} ord={ord(ch):>7} U+{ord(ch):04X} {unicodedata.name(ch)}")
print()
print("chr(235) :", chr(235))
print("chr(ord('ë')) == 'ë' :", chr(ord("ë")) == "ë")
print("\\u00eb :", "ë")
print("\\U0001f30d :", "\U0001f30d")
print("\\N{EURO SIGN} :", "\N{EURO SIGN}")
print("\\N{...} == chr(0xEB) :", "\N{LATIN SMALL LETTER E WITH DIAERESIS}" == chr(0xEB))
print("lookup('EURO SIGN') :", unicodedata.lookup("EURO SIGN"))
print("category / plane 🌍 :", unicodedata.category("🌍"), "/ plane", ord("🌍") // 0x10000)
print("ord('ë') vs its bytes :", ord("ë"), "vs", list("ë".encode("utf-8")))
try:
ord("ab")
except TypeError as e:
print("ord('ab') :", type(e).__name__ + ":", e)you see
A ord= 65 U+0041 LATIN CAPITAL LETTER A
ë ord= 235 U+00EB LATIN SMALL LETTER E WITH DIAERESIS
€ ord= 8364 U+20AC EURO SIGN
🌍 ord= 127757 U+1F30D EARTH GLOBE EUROPE-AFRICA
chr(235) : ë
chr(ord('ë')) == 'ë' : True
\u00eb : ë
\U0001f30d : 🌍
\N{EURO SIGN} : €
\N{...} == chr(0xEB) : True
lookup('EURO SIGN') : €
category / plane 🌍 : So / plane 1
ord('ë') vs its bytes : 235 vs [195, 171]
ord('ab') : TypeError: ord() expected a character, but string of length 2 found
ordtakes exactly one character.ord("ab")raisesTypeError, because a string of two has two answers.chr(0x110000)raisesValueError. The code space really does stop atU+10FFFF; there is no character above it to name.- Raw strings kill the escapes.
r"\u00eb"is six literal characters, which is occasionally what you want and usually a bug. unicodedata.nameraisesValueErrorfor characters that have no name — every control character, newline included.- Reaching for
.encode()to "get a character's number" gives you bytes, not the code point.ord("ë")is 235; its UTF-8 bytes are 195 and 171. "\N{...}"needs the name in full and in capitals.\N{euro sign}works,\N{EURO}does not.
Now the decoupling, which runs along two axes. The first axis is the one this whole chapter builds toward: the same code point can be serialised into different byte sequences by different encodings. U+00EB is [EB] in Latin-1, [C3 AB] in UTF-8, [EB 00] in UTF-16LE. One identity, three byte spellings. That is section 3's business — identity versus storage.
The second axis is subtler, and it's the one that produces the failed ==. The same visible character can be built from different sequences of code points. Your ë might be the single precomposed U+00EB. Or it might be two code points glued together: U+0065 (plain e) followed by U+0308 (COMBINING DIAERESIS, the two dots). Both render as an identical ë on screen, but they are different strings — one code point long versus two. So 'ë' == 'ë' can be False. This is identity (code points) diverging from appearance (glyph).
import unicodedata
a = "ë" # precomposed: one code point
b = "ë" # decomposed: 'e' + combining diaeresis
print(len(a), len(b), a == b) # 1 2 False — same glyph, different identity
print(unicodedata.normalize("NFC", b) == a) # True — NFC folds them togetherunicodedata.normalize('NFC', s) folds a string to its composed canonical form; 'NFD' gives the decomposed form. This is why any time you compare or store user-entered text you normalize first — otherwise two names that look identical to a human are two different keys to your database. The rule to carry: code point ≠ byte sequence ≠ glyph. Three layers, one standard pinning down only the top one.
ë becomes e plus U+0308. macOS filenames arrive like this, which is why a name that looks right can still miss.Ａ to A, the ligature fi to fi. Useful for search, lossy for storage.ß to ss and handles the Greek final sigma, which lower() deliberately does not."straße".upper() is "STRASSE", and it does not round-trip — uppercasing is not reversible in every language.you type
# compare_text.py -- why two identical-looking strings fail ==, and the fix
import unicodedata
a = "ë" # ë precomposed -> 1 code point
b = "ë" # ë e + COMBINING DIAERESIS -> 2 code points
print("a =", a, "| len", len(a), "|", [f"U+{ord(c):04X}" for c in a], "|", a.encode("utf-8").hex(" "))
print("b =", b, "| len", len(b), "|", [f"U+{ord(c):04X}" for c in b], "|", b.encode("utf-8").hex(" "))
print("a == b :", a == b)
print("NFC(b) == a :", unicodedata.normalize("NFC", b) == a)
print("NFD(a) == b :", unicodedata.normalize("NFD", a) == b)
print("len NFC / NFD :", len(unicodedata.normalize("NFC", b)), "/", len(unicodedata.normalize("NFD", a)))
print()
print("'straße'.upper() :", "straße".upper())
print("upper == 'STRASSE' :", "straße".upper() == "STRASSE")
print("'straße'.casefold():", "straße".casefold())
print("casefold match :", "straße".casefold() == "STRASSE".casefold())
print("'İ'.lower() :", repr("İ".lower()), "len", len("İ".lower()))
print("'ẞ'.lower() :", repr("ẞ".lower()))
def key(t): # the one function to compare user text with
return unicodedata.normalize("NFC", t).casefold()
print()
print("key(b) == key('Ë') :", key(b) == key("Ë"))
print("naive b == 'Ë' :", b == "Ë")you see
a = ë | len 1 | ['U+00EB'] | c3 ab
b = ë | len 2 | ['U+0065', 'U+0308'] | 65 cc 88
a == b : False
NFC(b) == a : True
NFD(a) == b : True
len NFC / NFD : 1 / 2
'straße'.upper() : STRASSE
upper == 'STRASSE' : True
'straße'.casefold(): strasse
casefold match : True
'İ'.lower() : 'i̇' len 2
'ẞ'.lower() : 'ß'
key(b) == key('Ë') : True
naive b == 'Ë' : False
'ë' == 'ë'can beFalse. They render identically; one is a single code point, the other is two.- Length changes with form. The decomposed
ëhaslen2 and three UTF-8 bytes; the composed one haslen1 and two bytes. casefold()is notlower().'straße'.lower()is unchanged; onlycasefoldgives you'strasse'.- Never store NFKC output as the user's data. It rewrites
½to1⁄2and cannot be undone — use it for a search key, beside the original. 'İ'.lower()is two code points. Case is a language-dependent mapping, not a per-character flip.- Normalising your query but not your stored keys fixes nothing. Both sides of every comparison go through the same
key().
chr(ord(c)) == c and ord(chr(n)) == n. They are the two directions of the same lookup table — no encoding involved, no bytes involved, just the identity map. If you ever catch yourself reaching for .encode() to "get a character's number," stop: that's ord's job, and it never touches storage.str is code-point based, so a single astral character like '🌍' has len exactly 1, and ord('🌍') is 0x1F30D, well above U+FFFF. JavaScript, whose strings are UTF-16 under the hood, reports that same globe as length 2 — because it counts the two surrogate halves. Same character, same identity, different counting — a direct consequence of one language exposing code points and the other exposing storage units. Section 6 pays this off.Identity is settled: ë is U+00EB, everywhere, forever. But how does U+00EB become the bytes C3 AB on disk? That number appears nowhere in 0xEB. Time to open the black box and derive UTF-8 one bit at a time. →
03UTF-8, decoded bit by bit
You accept that ë is U+00EB. But 'ë'.encode('utf-8') handed you b'\xc3\xab', and 0xEB is neither of those numbers. Where did C3 and AB come from? Until you can produce those two bytes from 0xEB with a pencil, UTF-8 stays magic. You'll never be able to look at a truncated multibyte sequence and know what broke. So we're going to turn the black box into an algorithm you run on paper. Give me a code point, and I'll give you the exact bytes and tell you why every single bit sits where it does.
UTF-8 is a variable-width encoding. It packs a code point's bits into 1, 2, 3, or 4 bytes, using self-describing leading bytes. The only question it asks is this: how many significant bits does this code point need? The four ranges map to four templates, where the x's are payload slots you fill with the code point's bits:
U+0000 .. U+007F (≤ 7 bits) 0xxxxxxx # 1 byte — identical to ASCII
U+0080 .. U+07FF (≤11 bits) 110xxxxx 10xxxxxx # 2 bytes
U+0800 .. U+FFFF (≤16 bits) 1110xxxx 10xxxxxx 10xxxxxx # 3 bytes
U+10000..U+10FFFF (≤21 bits) 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx # 4 bytesTwo invariants make this robust. First: a leading byte's high bits announce the total length. 0 means 1 byte, 110 means 2, 1110 means 3, 11110 means 4. Second: every continuation byte matches 10xxxxxx, and nothing else does. A continuation byte can never be mistaken for a start byte, and a start byte can never be mistaken for a continuation. Hold onto that — it's the whole reason UTF-8 won.
Now let's derive ë by hand. ë = U+00EB = 235 = binary 1110 1011, which is 8 significant bits. Eight is more than 7, so the 1-byte form won't hold it; we go to the 2-byte form, which offers 11 payload slots. Left-pad our value to 11 bits: 000 1110 1011. Split it 5 + 6: 00011 and 101011. Drop those into the two templates:
U+00EB = 235 = 000 1110 1011 # 11 payload bits: 00011 | 101011
byte 1: 110 00011 = 1100 0011 = 0xC3 # 110 marker + first 5 payload bits
byte 2: 10 101011 = 1010 1011 = 0xAB # 10 marker + last 6 payload bits
result: b'\xc3\xab' # exactly what Python produced ✓There it is: C3 AB, derived from 0xEB with nothing but bit-shuffling. The C3 and AB aren't stored anywhere in the number. They're the number's bits re-packaged into a self-describing frame. Run 'ë'.encode('utf-8').hex() and Python confirms 'c3ab'.
U+00EB into a 110xxxxx lead byte and a 10xxxxxx continuation byte — five bits then six — yielding 0xC3 0xAB.Two invariants buy you UTF-8's killer property: self-synchronization. Drop into a UTF-8 stream at any random byte and you can find a character boundary instantly. If the byte starts with 10, you're mid-character — step backward until you hit a byte that doesn't. If it starts with 0 or 11, you're already at a character start. Because no continuation byte can masquerade as a start, a single corrupted byte damages exactly one character — the rest of the file stays perfectly readable. UTF-16 has no such property; one lost byte shifts every subsequent pair and shreds the entire remainder. Self-sync is why UTF-8 became the encoding of the web.
The 1-byte branch hides a gift. For There are only three cliffs in all of Unicode — at U+0000 through U+007F the template is 0xxxxxxx — the top bit is zero and the low 7 bits are the ASCII value. So UTF-8 is byte-for-byte identical to ASCII for the first 128 characters. Every ASCII file ever written is already valid UTF-8. That backward compatibility is the second reason UTF-8 won: the entire existing corpus came along for free.
U+0080, U+0800, and U+10000. They are the only places the price changes. Slide across one and every character beyond it silently costs one more byte.A, 4 for an emoji — so the very same sentence can weigh wildly different amounts on disk depending only on its script.
0xxxxxxx → a lone ASCII character. 110xxxxx → "a 2-byte character starts here." 1110xxxx → 3-byte start. 11110xxx → 4-byte start. 10xxxxxx → "I am a fragment; a start byte is behind me." This is the whole grammar. Memorise these five shapes and a hex dump stops being a wall of numbers.U+10FFFF — about 1.1 million points — because of UTF-8's 4-byte ceiling? Not quite: the ceiling was fixed by UTF-16, which can only address 21 bits' worth of code points through its surrogate mechanism. UTF-8 could technically encode up to 6 bytes and billions of points, but it was deliberately capped at 4 bytes / U+10FFFF to stay interchangeable with UTF-16. The smaller sibling set the limit for the whole family.You can now turn any code point into bytes by hand. But you rarely call .encode directly — you call open(), print(), socket.recv(). Next: every one of those is encode or decode wearing a costume. →
# byte_surgeon.py -- open a text file as raw bytes and operate on it with a scalpel
from pathlib import Path
P = Path("tracks.txt")
TEXT = "Blinding Lights\nBjörk — Jóga\nさくら\n"
P.write_text(TEXT, encoding="utf-8", newline="\n") # chapter 15's writer, codec named
raw = P.read_bytes() # the 'rb' door: no codec at all
txt = P.read_text(encoding="utf-8") # the decode door
print("bytes on disk :", len(raw))
print("code points in RAM:", len(txt))
print("the accent tax :", len(raw) - len(txt), "extra bytes")
needle = "ö"
nb = needle.encode("utf-8")
i = raw.find(nb) # search BYTES for BYTES
print()
print(f"'{needle}' encodes to :", nb.hex(" "), f"({len(nb)} bytes)")
print("byte offset :", i, "| str index", txt.index(needle))
print("raw[i-2:i+3] :", raw[i - 2:i + 3], "->", raw[i - 2:i + 3].decode("utf-8"))
half = raw[:i + 1] # a cut through the middle of ö
print()
print("cut after 1 of 2 :", half[-3:].hex(" "))
try:
half.decode("utf-8")
except UnicodeDecodeError as e:
print("strict decode :", type(e).__name__ + ":", e.reason, "at byte", e.start)
print("errors='replace' :", repr(half[-4:].decode("utf-8", errors="replace")))
print()
for n, line in enumerate(txt.splitlines(), 1):
print(f"line {n}: {len(line):>2} code points | {len(line.encode('utf-8')):>2} utf-8 bytes | {line}")
P.unlink()
bytes on disk : 43 code points in RAM: 33 the accent tax : 10 extra bytes 'ö' encodes to : c3 b6 (2 bytes) byte offset : 18 | str index 18 raw[i-2:i+3] : b'Bj\xc3\xb6r' -> Björ cut after 1 of 2 : 42 6a c3 strict decode : UnicodeDecodeError: unexpected end of data at byte 18 errors='replace' : '\nBj�' line 1: 15 code points | 15 utf-8 bytes | Blinding Lights line 2: 12 code points | 16 utf-8 bytes | Björk — Jóga line 3: 3 code points | 9 utf-8 bytes | さくら
byte_surgeon.py and run it. We open one file through both doors and let the answers argue. The text door reports 33 code points; the raw door reports 43 bytes. That ten-byte gap is the accent tax, and you can now point at every byte of it. Then watch the offsets. ö sits at byte 18 and at string index 18, which is a coincidence, not a law. Everything before it on those two lines is ASCII, so the two rulers happen to agree. Put an accent in line 1 and they part company immediately, and stay apart by one per extra byte, forever. Next we cut. raw[:i+1] keeps the first half of ö and throws the second away, and the strict decoder refuses it: unexpected end of data. Read that message precisely, because it is not saying the bytes are wrong. It is saying the read stopped in the middle of a character — exactly what a chunked socket read does at 4096 bytes, and exactly why the decoder behind open() is incremental. The last block is the honest summary: さくら is 3 code points and 9 bytes, because every character in the Japanese kana block costs three. Two things to try. Change TEXT so line 1 opens with ö, then watch raw.find and txt.index disagree. Then delete the newline="\n" argument on Windows and run it again — the byte count climbs by three, one \r per line, while the code point count never moves. (That keyword needs Python 3.10 or newer; on older versions use open() with the same argument.)04encode and decode are the only doors
You've read files, printed API responses, and written logs for years without once consciously choosing an encoding. So when I claim that every read is a decode and every write is an encode, it sounds like an abstraction you can safely skip. You can't. It means those two functions from section 1 aren't niche utilities. They are the only hinges the entire text/binary boundary swings on. open(), print(), sockets, JSON over HTTP, subprocess output — all of it is str.encode and bytes.decode in costume. Once you can point at the door in any I/O situation, encoding bugs stop being spooky. They collapse into one question: which of my two doors used the wrong key?
There are exactly two functions that cross the wall, and each takes a codec name. str.encode(codec, errors) walks the code points and emits a bytes. bytes.decode(codec, errors) walks the bytes and reconstructs the code points. Everything else is a wrapper around these two. Consider open:
with open("note.txt", "w", encoding="utf-8") as f:
f.write("héllo") # encode door: str → bytes → write() syscall
with open("note.txt", "rb") as f:
print(f.read()) # b'h\xc3\xa9llo' — raw bytes, no codec at allopen(path, 'r', encoding='utf-8') returns an io.TextIOWrapper layered over a raw binary buffer. On .read() it pulls bytes from the OS via the read() syscall and feeds them through the codec's incremental decoder. On .write() it runs the incremental encoder and hands the resulting bytes to write(). The wrapper is nothing but an automated decode/encode door with a buffer bolted on. Open the same file 'rb' and you skip the wrapper entirely. You get raw bytes, no codec, no interpretation. Sockets and pipes are always bytes. sock.recv() returns bytes and you decode. sock.sendall() demands bytes, so you encode. There is no text on a wire, ever.
Why incremental? Because a multibyte character can straddle a buffer boundary. Say a chunked read splits ï = C3 AF between two 4096-byte reads. The C3 lands at the end of one chunk, and the AF at the start of the next. A naive decoder would choke on the lone C3. The incremental decoder instead holds the partial byte, emits nothing, and completes the character when AF arrives. This is why streaming text is a codec object with state, not a one-shot function call.
str lives inside; every file, socket, and pipe outside speaks bytes — encode and decode are the doors, and open(…, encoding=…) just bolts an automatic codec onto them.The codec name is itself a key into the codecs registry. codecs.lookup('utf-8') resolves to an encoder/decoder pair. 'utf-8', 'latin-1', 'ascii', 'utf-16', 'cp1252' each name a different pair. And the errors parameter governs what happens when the codec meets input it can't map. This matters more than it looks:
b = b"\xff\xfehi" # \xff is not valid UTF-8
print(b.decode("utf-8", errors="replace")) # '��hi' — U+FFFD for each bad byte
print(b.decode("utf-8", errors="ignore")) # 'hi' — bad bytes dropped
print(b.decode("utf-8", errors="backslashreplace")) # '\xff\xfehi'
# errors="strict" (the default) would raise UnicodeDecodeError here'strict' (the default) raises UnicodeDecodeError / UnicodeEncodeError. 'replace' substitutes U+FFFD (the � replacement character) on decode, or '?' on encode. 'ignore' silently drops the offender. 'backslashreplace' emits \xNN escapes. And 'surrogateescape' is the clever one. It smuggles each undecodable byte into the reserved surrogate range U+DC80–U+DCFF. So a later encode with the same handler can round-trip it back to the exact original byte. That's how os.fsdecode survives a filename on disk that isn't valid UTF-8. The raw bytes ride through your program hidden in surrogates and come out untouched.
U+FFFD, the � you have seen on a thousand web pages. The damage is visible and countable — text.count("\ufffd") is a data-quality metric.\xc3, four characters you can read and grep. Ideal for a log line about bad input.U+DC80–U+DCFF. Encode with the same handler and the original byte comes back exactly.UnicodeDecodeError carries .object, .start, .end and .reason. That is enough to print the offending bytes and the position, which is what a good error message does.you type
# errors_tour.py -- four answers to one question: what do I do with a byte I cannot read
s = "Zoë Müller"
full = s.encode("utf-8")
cut = full[:7] # a chunked read that stopped mid-character
print("full :", full.hex(" "), f"({len(full)} bytes)")
print("cut :", cut.hex(" "), f"({len(cut)} bytes) <- ends on a lone c3")
print()
try:
cut.decode("utf-8") # errors='strict' is the default
except UnicodeDecodeError as e:
print("strict :", type(e).__name__)
print(" reason :", e.reason)
print(" bytes", e.start, "..", e.end, ":", e.object[e.start:e.end])
print("replace :", repr(cut.decode("utf-8", errors="replace")))
print("ignore :", repr(cut.decode("utf-8", errors="ignore")))
print("backslashreplace:", repr(cut.decode("utf-8", errors="backslashreplace")))
sur = cut.decode("utf-8", errors="surrogateescape")
print("surrogateescape :", repr(sur))
print(" round-trips :", sur.encode("utf-8", errors="surrogateescape") == cut)
print()
print("encode side, strict :", end=" ")
try:
print("naïve".encode("ascii"))
except UnicodeEncodeError as e:
print(type(e).__name__ + ":", e.reason, "at", e.start)
print("encode replace :", "naïve".encode("ascii", errors="replace"))
print("encode xmlcharref :", "naïve".encode("ascii", errors="xmlcharrefreplace"))
print("encode namereplace :", "naïve".encode("ascii", errors="namereplace"))you see
full : 5a 6f c3 ab 20 4d c3 bc 6c 6c 65 72 (12 bytes)
cut : 5a 6f c3 ab 20 4d c3 (7 bytes) <- ends on a lone c3
strict : UnicodeDecodeError
reason : unexpected end of data
bytes 6 .. 7 : b'\xc3'
replace : 'Zoë M�'
ignore : 'Zoë M'
backslashreplace: 'Zoë M\\xc3'
surrogateescape : 'Zoë M\udcc3'
round-trips : True
encode side, strict : UnicodeEncodeError: ordinal not in range(128) at 2
encode replace : b'na?ve'
encode xmlcharref : b'naïve'
encode namereplace : b'na\\N{LATIN SMALL LETTER I WITH DIAERESIS}ve'
errors="ignore"is the handler that makes a bug invisible. It is almost never the right answer, and it is the most-copied one.'replace'meansU+FFFDon decode and'?'on encode. One name, two different substitutes, depending on the direction.- A surrogate-escaped string only survives
encodewith the same handler. A plainsur.encode("utf-8")raisessurrogates not allowed. xmlcharrefreplaceandnamereplaceare encode-only. Asking for them on a decode raisesTypeErrorfrom inside the error callback.- The handler applies to the codec, not the file.
open(…, errors="replace")hides bad bytes in every read from that handle, including the ones you cared about. - Counting
�after a lossy decode is a habit worth having. A silent zero is a good file; a non-zero count is a conversation with whoever sent it.
The law that ties it together: for any str s and any codec that covers it, s.encode(codec).decode(codec) == s. The doors are exact inverses when you use the same key on both sides. Encode with UTF-8, decode with UTF-8 — you get s back, guaranteed. Encode with one key and decode with another, and you get section 5's nightmare.
open(path, encoding='latin-1') reads a UTF-8 file, it doesn't error — it happily runs the wrong decode door and hands you a plausible-looking, silently-wrong string. The wrapper hides the codec, which is exactly why encoding bugs feel like they come from nowhere. Every open(), every .decode(), every .encode() is a door with a key slot. Name the key, every time.'latin-1' is the one codec that can never fail in either direction. On decode, Latin-1 maps byte N to U+00NN for all 256 bytes — every byte is legal. On encode, every code point U+0000–U+00FF maps straight back to one byte. That total, lossless coverage of raw bytes makes 'latin-1' the go-to for reading arbitrary binary as if it were text and re-encoding it unchanged — and, as the next section shows, the exact reason it silently produces garbage instead of crashing.Same key both sides = round-trip. Different keys = the most infamous bug in all of text handling — the one that works on your laptop and corrupts on your coworker's. Let's dissect it. →
05Mojibake and 'works on my machine'
This is the bug that gets junior engineers blamed. Your script reads a CSV of customer names flawlessly on your Mac. You ship it. On your coworker's Windows box every ë is now ë, or worse, the program dies with UnicodeDecodeError on line 4,213. Not one character of your code changed. It feels like randomness, or a Python bug, or cosmic punishment. It is none of those. It is a decoder reading correct bytes against the wrong table, and the reason it differs by machine is that open() silently picked a locale-dependent default encoding. Let's take it apart mechanically, then kill the entire bug class with one habit.
Mojibake is a decode-side type confusion: correct bytes, wrong codec. Take ë stored as UTF-8, which section 3 taught you is C3 AB. Now decode those two bytes as Latin-1, where by definition every byte is its own character — byte N maps to U+00NN. You get U+00C3 + U+00AB = 'Ã' + '«' = 'ë'. The bytes were never damaged. They were read against the wrong table, so a group of two bytes that meant one character got misread as two separate one-byte characters:
s = "café — 30€"
b = s.encode("utf-8") # the correct bytes on disk
print(b.decode("utf-8")) # 'café — 30€' — right key, perfect
print(b.decode("latin-1")) # 'café — 30€' — wrong key, mojibakeThe reverse direction fails harder, and understanding why is the key to the whole asymmetry. Many byte sequences are simply illegal UTF-8. A lone continuation byte like 0xAB (which is 10101011 — a 10xxxxxx fragment) with no leading byte in front of it matches no valid template from section 3. So the strict UTF-8 decoder can't guess; it raises UnicodeDecodeError: invalid start byte at a precise position. This is the asymmetry: Latin-1 can decode any byte (all 256 are legal), UTF-8 cannot. So UTF-8 text mistakenly read as Latin-1 tends to silently garble, while Latin-1 text mistakenly read as UTF-8 tends to loudly crash:
bad = "ñ".encode("latin-1") # b'\xf1' — a lone high byte
try:
bad.decode("utf-8")
except UnicodeDecodeError as e:
print(e.reason, e.start, e.object) # 'invalid start byte' 0 b'\xf1'C3 AB is ë under UTF-8 but mojibake under Latin-1, and an invalid lead byte raises UnicodeDecodeError — so always name the encoding.Now the "works on my machine" part. When you call open(path) in text mode with no encoding= argument, historically Python used locale.getpreferredencoding(False) to pick the codec. On modern macOS and Linux that resolves to UTF-8. On Western-configured Windows it was cp1252 (a Latin-1 superset); on a Japanese Windows box it might be cp932. Same bytes, same code, different default table per machine — therefore different result. Your laptop and your coworker's laptop literally ran two different decoders behind the identical line open('names.csv'). Nothing was random; the codec was just invisible, and its value came from the operating system's locale.
Python surfaced this landmine with EncodingWarning (PEP 597). Run with python -X warn_default_encoding and the interpreter flags every open() and .decode() that leaned on the locale default — a lint for exactly this bug. The fix, though, isn't a library or a flag. It's a discipline: name the codec on every boundary crossing. Write open(path, encoding='utf-8') and b.decode('utf-8') everywhere. The moment the codec is explicit, the table no longer depends on whose machine runs the code, and the entire bug class evaporates.
Traceback (most recent call last):
File "load_members.py", line 7, in <module>
for row in open("members.csv", encoding="utf-8"):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xeb in position 12: invalid continuation byte0xeb is 1110 1011 — by section 03's table a perfectly legal three-byte lead, so the decoder committed to reading three bytes. The next byte was 0x20, a space, and a space is not 10xxxxxx. That is why the reason says continuation, and why it blames position 12, where the character began, rather than 13, where it fell apart. It is really saying: whoever wrote this file used a one byte per character table, where 0xeb means ë all by itself and nothing follows it. Section 04's card named the two reasons you meet most; this is the third, and it is the most specific of the three — it tells you the file is nearly right, that a lead byte was found and only its follower was wrong. Nothing is broken in your loop, and the line number is a red herring: the file was already wrong before your program opened it.latin-1 never raises, so switching to it always "works" and tells you nothing about whether it is right; read the bytes once and try utf-8 then cp1252 strictly, and record which one wonTraceback (most recent call last):
File "write_report.py", line 7, in <module>
f.write(f"{title}\t{plays}\n")
File "cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-2: character maps to <undefined>open(path, "w") and named no encoding, so the machine chose one — and you can see which, in the second frame: cp1252. That frame is the only place the codec was ever written down, which is exactly why this bug feels like it arrived from nowhere. charmap is that codec family's name for itself. It is really saying: you handed me three characters — さくら, position 0 to 2 of that one write — and my table has no slot for any of them, and I will not invent one. Now flinch at the part the traceback does not mention. report.txt is already on disk, holding 22 bytes — the first row, plus the \r\n Windows swapped in for its newline. That row encoded fine, and the closing with flushed it on the way out. The job died halfway and left behind a file that looks finished.open(path, "w", encoding="utf-8"); a file written under one machine's locale table is the next machine's UnicodeDecodeErrorTraceback (most recent call last):
File "count_plays.py", line 7, in <module>
for line in raw.split("\n"):
^^^^^^^^^^^^^^^
TypeError: a bytes-like object is required, not 'str'raw came from read_bytes() — the rb door, the one with no codec behind it at all. So raw is a run of integers and "\n" is a code point, and there is nothing in between the two to translate. It is really saying: you are one decode short, and you are standing on the wrong side of the wall to be using string tools. Then watch out for the fix that makes the message go away. raw.split(b"\n") runs perfectly, and hands you a dict keyed by b'Bj\xc3\xb6rk' — so the report you write tomorrow asks plays.get("Björk"), gets None, and prints a zero. No exception, no traceback, no clue. Silencing a wall error by moving everything into bytes is how a loud bug becomes a quiet one.text = raw.decode("utf-8") on the line after the read — and let every line below it work in str# mojibake_lab.py -- make the bug on purpose, then undo it
ORIGINAL = "Zoë Müller — café 30€"
disk = ORIGINAL.encode("utf-8") # what a correct writer left on disk
print("on disk :", disk.hex(" "))
print("bytes/points :", len(disk), "/", len(ORIGINAL))
print()
good = disk.decode("utf-8") # right key
bad = disk.decode("cp1252") # wrong key, deliberately
raw = disk.decode("latin-1") # wrong key, the raw-byte variant
print("utf-8 :", good)
print("cp1252 :", bad)
print("latin-1 :", repr(raw))
print("bytes intact :", good.encode("utf-8") == disk)
print()
repaired = bad.encode("cp1252").decode("utf-8") # undo the wrong read, redo it right
print("repaired :", repaired)
print("== ORIGINAL :", repaired == ORIGINAL)
print()
for codec in ("utf-8", "utf-16", "cp1252", "latin-1"):
try:
assert ORIGINAL.encode(codec).decode(codec) == ORIGINAL
print(f"round-trip {codec:<8}: OK, {len(ORIGINAL.encode(codec)):>2} bytes")
except UnicodeEncodeError as e:
print(f"round-trip {codec:<8}: cannot encode {e.object[e.start:e.end]!r}")
for probe in (b"\xf1", b"\xab"): # the other direction: it crashes
try:
probe.decode("utf-8")
except UnicodeDecodeError as e:
print(f"{probe} as utf-8 :", e.reason)
on disk : 5a 6f c3 ab 20 4d c3 bc 6c 6c 65 72 20 e2 80 94 20 63 61 66 c3 a9 20 33 30 e2 82 ac bytes/points : 28 / 21 utf-8 : Zoë Müller — café 30€ cp1252 : Zoë Müller — café 30€ latin-1 : 'Zoë Müller â\x80\x94 café 30â\x82¬' bytes intact : True repaired : Zoë Müller — café 30€ == ORIGINAL : True round-trip utf-8 : OK, 28 bytes round-trip utf-16 : OK, 44 bytes round-trip cp1252 : OK, 21 bytes round-trip latin-1 : cannot encode '—' b'\xf1' as utf-8 : unexpected end of data b'\xab' as utf-8 : invalid start byte
mojibake_lab.py and run it. The bug is easier to trust once you have built it on purpose. One string, encoded once, decoded three ways. With the right key you get Zoë Müller — café 30€ back. With cp1252 you get the classic garble, and with latin-1 you get the same garble plus two control characters that your terminal cannot even draw — which is why we print that one with repr. Now the line that matters: bytes intact: True. The twenty-eight bytes on disk are identical in all three cases. Nothing was corrupted, nothing was lost, and the file is fine. Only the reader was wrong. That is why the repair works at all. bad.encode("cp1252") undoes the wrong decode and hands back the original bytes, and .decode("utf-8") then reads them properly. The round-trip table underneath makes the law concrete. UTF-8, UTF-16 and cp1252 all round-trip this string, at 28, 44 and 21 bytes; latin-1 cannot even encode the em dash, so it fails on the way out rather than the way back. And the last two lines name the asymmetry precisely: \xf1 announces a four-byte character and then the data ends, while \xab is a continuation byte with nothing in front of it. Unexpected end of data and invalid start byte are two different diagnoses, and section 3's templates are why. Two things to try. Change the wrong key from cp1252 to cp437 and watch a completely different garble appear from the same bytes. Then encode as utf-16 and decode as utf-8, and notice that this one crashes rather than garbling — UTF-16 sprays zero bytes that UTF-8 has no template for.encoding="utf-8" to every open(), and name the codec on every .encode() / .decode(). Never rely on the locale default — it is the single variable that differs between your machine and everyone else's. Python 3.15's UTF-8-mode-by-default (PEP 686) finally makes UTF-8 the default, but explicit-encoding stays the portable habit for any code that must also run on older interpreters. Explicit beats implicit precisely because the implicit value is set by someone else's operating system.✗ The myth
"The file got corrupted / Python is buggy / the coworker's machine is broken — the characters came out mangled."
✓ The reality
The bytes on disk are byte-for-byte identical everywhere. A decoder read them against the wrong table because open() chose a locale-dependent codec. Pass encoding="utf-8" on both the write and the read and the same bytes decode identically on every OS.
You can now diagnose the byte-level bug. One last trap lives at the other end — not in the bytes, but in counting. Ask a string "how long are you?" and it answers a question you probably didn't mean to ask. →
Zoë Müller — café 30€. No file, no code, no traceback. First, read it backwards. Every accented character became exactly two characters, and the first of each pair is à or â. Ask yourself what byte à is in a single-byte table, and what a UTF-8 lead byte looks like — that pairing is the fingerprint, and it names both keys without you guessing. Second, write unmojibake(text, wrong, right). Two calls, no loops: encode the garbled text with the codec that misread it, then decode the recovered bytes with the codec that wrote it. Print the hex of the result and check it against the twenty-eight bytes the last program showed you. Third, make it safe to run twice. Wrap it so text that is already correct comes back untouched: try cp1252, then latin-1, and return the input unchanged if both raise. Test it on "café", on "café", and on "éè", and say in one line why the last two are refused. One hint and no more: — encodes to e2 80 94, and cp1252 maps 0x80 to € while latin-1 leaves it an unprintable control character. That single byte is how you tell the two wrong keys apart.show the solution
# unmojibake.py -- you are handed the garble; recover the original
GARBLED = "Zoë Müller — café 30€" # copied out of the broken report
def unmojibake(text, wrong="cp1252", right="utf-8"):
"""Undo the wrong decode, then redo it with the right key."""
return text.encode(wrong).decode(right)
def repair(text, wrongs=("cp1252", "latin-1")):
for w in wrongs:
try:
return w, unmojibake(text, w)
except (UnicodeEncodeError, UnicodeDecodeError):
continue
return None, text # not mojibake, or not this flavour
wrong, fixed = repair(GARBLED)
print("garbled :", GARBLED)
print("undone as :", wrong)
print("fixed :", fixed)
print("bytes now :", fixed.encode("utf-8").hex(" "))
print()
print("lengths :", len(GARBLED), "->", len(fixed), "code points")
print("check :", fixed == "Zoë Müller — café 30€")
print()
for sample in ("café", "café", "éè"): # the guard: never repair correct text
w, out = repair(sample)
print(f"{sample!r:<12} -> undone as {str(w):<8} {out!r}")
# ------------- what it prints -------------
garbled : Zoë Müller — café 30€
undone as : cp1252
fixed : Zoë Müller — café 30€
bytes now : 5a 6f c3 ab 20 4d c3 bc 6c 6c 65 72 20 e2 80 94 20 63 61 66 c3 a9 20 33 30 e2 82 ac
lengths : 28 -> 21 code points
check : True
'café' -> undone as cp1252 'café'
'café' -> undone as None 'café'
'éè' -> undone as None 'éè'
# Why the guard refuses the last two:
# 'café'.encode('cp1252') is b'caf\xe9'; \xe9 announces a 3-byte UTF-8 character
# and the data ends -> UnicodeDecodeError, so repair() returns the text unchanged.
# The guard is a heuristic, not a proof. The real fix is always at the source:
# name encoding='utf-8' on the write AND the read, and the garble never happens.06len() measures code points, not width
You build a 280-character tweet limiter. Or you truncate a name to fit a VARCHAR(20) column. Or you draw a box around a string in the terminal. And it's subtly, maddeningly wrong. len('🇫🇷') returns 2 for a single flag. A comment with a family emoji counts as 7 when the user typed one "character." Truncating with s[:20] slices an emoji into rubble �. The root cause is that len() answers exactly one specific question: how many code points. That number is almost never the same as "how many bytes on disk" or "how many symbols a human sees." Knowing which of three counts your task actually needs is what prevents an entire family of off-by-one, overflow, and truncation bugs.
len(s) on a str returns the number of code points. Internally that's PyUnicode_GET_LENGTH, an O(1) field read of the code-point array's length. Not a byte count. Not a glyph count. There are three genuinely different measurements, and different text tasks demand different ones:
(1) Byte count = len(s.encode('utf-8')) — what a file, a socket, or a byte-sized database column actually consumes. A single ё is 1 code point but 2 UTF-8 bytes. A 🌍 is 1 code point but 4 bytes. (2) Code point count = len(s) — the string's own element count, and what indexing and slicing operate on. (3) Grapheme count = the number of user-perceived characters, defined by the Unicode text-segmentation algorithm UAX #29. The standard library does not compute this. You need a grapheme library that implements extended grapheme clusters.
for s in ["A", "ё", "🌍", "🇫🇷", "👨👩👧"]:
print(len(s.encode("utf-8")), len(s), "→", s)
# bytes code points glyph
# 1 1 A plain ASCII
# 2 1 ё one code point, two bytes
# 4 1 🌍 astral: one code point, four bytes
# 8 2 🇫🇷 one flag = TWO regional-indicator code points
# 18 5 👨👩👧 three people joined by two ZWJLook at how badly the three diverge, because several code points routinely render as one grapheme. A combining sequence 'e' + U+0308 is 2 code points / 1 grapheme. A country flag like 🇫🇷 is two Regional Indicator code points (U+1F1EB + U+1F1F7) / 1 grapheme / 8 UTF-8 bytes. A ZWJ emoji like 👨👩👧 is 5 code points — three people joined by two U+200D zero-width joiners — / 1 grapheme / 18 bytes. One symbol on screen, five elements in the array, eighteen bytes on the wire. Three completely different numbers for one "character."
len() counts code points only, so a flag emoji reads as 2 and a family as 5 — and slicing between a flag's two regional indicators breaks the glyph.rulers(s) returning three numbers for one string: UTF-8 bytes, code points, and code points after NFC. Run it over "Bonjour", "café", "ё", "🌍", "🇫🇷", the family emoji, and a decomposed ë written as "e\u0308". Predict each row before you run it, then check. Only one row changes under NFC, and you should be able to say which and why. Second, truncate to a byte budget without breaking a character. Write truncate_bytes(s, limit) that encodes, slices to limit bytes, and decodes with errors="ignore". Run it on "Zoë Müller" at 4, 5, 6 and 20 bytes. Explain in one line why budget 4 yields three characters and budget 5 yields four. Third, fix the é-comparison. Write same_text(x, y) that normalises both sides to NFC and casefolds both, then compare four pairs: the two ës, ë against Ë, "straße" against "STRASSE", and "café" against "cafe". Three should match and one should not; write the one-line reason the last pair is genuinely different text rather than a normalisation problem. One hint and no more: the two ës print identically in the output, so trust the numbers in the table and not your eyes.show the solution
# rulers.py -- measure text three ways, and compare it once, correctly
import unicodedata
SAMPLES = ["Bonjour", "café", "ё", "🌍", "🇫🇷", "👨👩👧", "ë"]
def rulers(s):
return len(s.encode("utf-8")), len(s), len(unicodedata.normalize("NFC", s))
print("string bytes points NFC points")
for s in SAMPLES:
by, cp, nf = rulers(s)
print(f"{s!r:<14}{by:>5}{cp:>8}{nf:>12}")
def truncate_bytes(s, limit, encoding="utf-8"):
"""Cut to a BYTE budget without ever emitting a half character."""
return s.encode(encoding)[:limit].decode(encoding, errors="ignore")
print()
name = "Zoë Müller"
for limit in (4, 5, 6, 20):
out = truncate_bytes(name, limit)
print(f"budget {limit:>2} bytes -> {out!r:<14} ({len(out.encode('utf-8'))} bytes used)")
def same_text(x, y):
"""The only comparison to trust on text a human typed."""
n = unicodedata.normalize
return n("NFC", x).casefold() == n("NFC", y).casefold()
print()
pairs = [("ë", "ë"), ("ë", "Ë"), ("straße", "STRASSE"), ("café", "cafe")]
for x, y in pairs:
print(f"{x!r:<10} vs {y!r:<10} == {x == y!s:<6} same_text {same_text(x, y)}")
# ------------- what it prints -------------
string bytes points NFC points
'Bonjour' 7 7 7
'café' 5 4 4
'ё' 2 1 1
'🌍' 4 1 1
'🇫🇷' 8 2 2
'👨\u200d👩\u200d👧' 18 5 5
'ë' 3 2 1
budget 4 bytes -> 'Zoë' (4 bytes used)
budget 5 bytes -> 'Zoë ' (5 bytes used)
budget 6 bytes -> 'Zoë M' (6 bytes used)
budget 20 bytes -> 'Zoë Müller' (12 bytes used)
'ë' vs 'ë' == False same_text True
'ë' vs 'Ë' == False same_text True
'straße' vs 'STRASSE' == False same_text True
'café' vs 'cafe' == False same_text False
# Read the output honestly, in two places:
# * the emoji rows do NOT line up -- f-string width pads by CODE POINTS,
# so a 2-point flag occupies far more screen than 2 columns. See the next box.
# * 'e\u0308' and '\u00eb' both print as ë. The bytes column is the only
# honest witness: 3 bytes and 2 points vs 2 bytes and 1 point.The danger is that str indexing and slicing work in code points. So s[:n] can cut straight through the middle of a grapheme. Truncate '🇫🇷' at [:1] and you keep only the first Regional Indicator. It renders as a bare regional letter, not a flag. Slice a database field at a byte budget and you can leave a half-encoded multibyte fragment that decodes to �. This is why a naive "characters remaining" counter or a naive truncation is a bug factory:
t = "Bonjour 🇫🇷"
print(len(t)) # 10 code points (8 letters + space + 2 indicators)
print(t[:9]) # 'Bonjour 🇫' — flag split, collapses to a lone letter
print([hex(ord(c)) for c in "🇫🇷"]) # ['0x1f1eb', '0x1f1f7'] — two code points, one flagOne clarification that ties this back to section 2: a single astral code point is one Python str element. len('🌍') == 1, because Python's str is code-point-based, not UTF-16-surrogate-based. JavaScript reports 2 for the same globe because its strings are UTF-16 and it counts the two surrogate halves. Python spares you that particular trap. But the flag and ZWJ traps remain, because those are about multiple code points forming one grapheme, which no code-point count can collapse.
#, then 0, then width, then grouping, then precision, then type. Swap any two and you get a ValueError, not a surprise.{:.>12} pads with dots; {:*^12} centres in stars.{:015,.2f} is sixteen characters wide, because the commas count towards the width.g). On a str it is a maximum length — a silent truncation, not an error.% multiplies by 100 and appends the sign, so 0.0725 is 7.2%. c is chr() in disguise: {:c} on 8364 prints €.f"{v:{w}.{p}f}" is how you build a table whose columns are computed, not typed.__format__. That is why datetime accepts %Y-%m-%d — the spec language is per-type, not global.you type
# formatspec.py -- the format mini-language, one row per knob
from datetime import datetime
name, v, n = "Zoë", 1234567.8915, 255
rows = [
("{:>12}", name), ("{:<12}", name), ("{:^12}", name),
("{:.>12}", name), ("{:*^12}", name), ("{:.2}", name),
("{:+.2f}", v), ("{: .2f}", v), ("{:,.2f}", v), ("{:_.2f}", v),
("{:015,.2f}", v), ("{:.3e}", v), ("{:.4g}", v),
("{:.1%}", 0.0725), ("{:#x}", n), ("{:#b}", 5), ("{:08b}", 5), ("{:c}", 8364),
]
for spec, val in rows:
print(f"{spec:<12}{val!r:<14} -> {spec.format(val)!r}")
print()
w, p = 14, 3
print("nested width ->", f"|{name:>{w}}|")
print("nested prec ->", f"{v:.{p}f}")
print("datetime spec ->", f"{datetime(2026, 8, 16, 21, 5):%Y-%m-%d %H:%M}")
print("conversions ->", f"{name!r} {name!s} {name!a}")
print("self-doc = ->", f"{v = :,.2f}")
print("literal brace ->", f"{{{name}}}")
print()
print("width counts CODE POINTS, not screen columns:")
for s in ("FR", "🇫🇷", "さくら"):
print(f" |{s:<8}| len {len(s)} bytes {len(s.encode('utf-8'))}")you see
{:>12} 'Zoë' -> ' Zoë'
{:<12} 'Zoë' -> 'Zoë '
{:^12} 'Zoë' -> ' Zoë '
{:.>12} 'Zoë' -> '.........Zoë'
{:*^12} 'Zoë' -> '****Zoë*****'
{:.2} 'Zoë' -> 'Zo'
{:+.2f} 1234567.8915 -> '+1234567.89'
{: .2f} 1234567.8915 -> ' 1234567.89'
{:,.2f} 1234567.8915 -> '1,234,567.89'
{:_.2f} 1234567.8915 -> '1_234_567.89'
{:015,.2f} 1234567.8915 -> '0,001,234,567.89'
{:.3e} 1234567.8915 -> '1.235e+06'
{:.4g} 1234567.8915 -> '1.235e+06'
{:.1%} 0.0725 -> '7.2%'
{:#x} 255 -> '0xff'
{:#b} 5 -> '0b101'
{:08b} 5 -> '00000101'
{:c} 8364 -> '€'
nested width -> | Zoë|
nested prec -> 1234567.891
datetime spec -> 2026-08-16 21:05
conversions -> 'Zoë' Zoë 'Zo\xeb'
self-doc = -> v = 1,234,567.89
literal brace -> {Zoë}
width counts CODE POINTS, not screen columns:
|FR | len 2 bytes 2
|🇫🇷 | len 2 bytes 8
|さくら | len 3 bytes 9
- Width pads by code points, never by screen columns. The flag row in the previous exercise is two "characters" wide and nothing lines up.
{:.2}on"Zoë"gives"Zo". Precision silently truncates strings, which is a real way to lose the end of a name.{:.1%}on0.0725prints7.2%, not7.3%. Python rounds half to even, in formatting as everywhere else.{:%}on the integer1gives100.000000%. Percent always multiplies by a hundred, whatever you meant.{name!a}gives'Zo\xeb'— ASCII-safe repr, escapes intact. Useful in a log you cannot guarantee is UTF-8.- A literal brace is doubled:
f"{{{name}}}"prints{Zoë}. There is no backslash escape for braces in an f-string.
VARCHAR(n), a packet size, an SMS segment — count bytes: len(s.encode('utf-8')). Algorithmic indexing and slicing inside your own code — count code points: len(s). Anything a human reads or edits — a "characters left" badge, cursor movement, truncating a display name — count graphemes: segment with a UAX #29 library first, then slice on grapheme boundaries. Using the wrong count is how one flag becomes a broken half-flag.🇫🇷 works by a rule with no analog in ordinary text: two Regional Indicator Symbol Letters, F and R, placed adjacent, are rendered by the font as the French flag. Put F next to I and you get the Finnish flag; put a lone F anywhere and you get a boxed letter F. There is no code point for "French flag" at all — the flag is an emergent grapheme from two country-code letters. That's why len insists it's two: at the code-point layer, it genuinely is.Text is now fully demystified: identity is a code point, storage is an encoding, the two doors are encode and decode, and length is three different questions. Next chapter carries the same wall to a bigger boundary — the file system itself, where those bytes finally hit the disk. →
- A
strand abytesare two types over two alphabets — about 1.1 million code points on one side, exactly 256 integers on the other — and Python 3 will never quietly turn one into the other, which is why"x" + b"y"stops you where Python 2 would have guessed. - A character has three independent layers: its identity is a code point that Unicode fixed once and forever (
ëis U+00EB, everywhere), its storage is whatever bytes an encoding chooses, and its appearance is a glyph — and a glyph can even be built from two code points, which is why two identical-lookingës can fail==until you normalise. - UTF-8 is a bit-packing rule, not a lookup table: the lead byte's high bits announce the length, every continuation byte is
10xxxxxxand nothing else ever is — which buys self-synchronisation (one bad byte damages one character) and makes the first 128 characters byte-identical to ASCII, the two reasons it won. str.encodeandbytes.decodeare the only two doors, andopen,print, sockets and JSON are those same two wearing costumes — so the whole law is same key both sides, and mojibake is not damage but correct bytes read against the wrong table by a decoder that had no way to know.len()answers exactly one question, how many code points, so pick your ruler by the job: bytes for anything with a storage or protocol budget, code points for indexing and slicing inside your own code, graphemes for anything a human counts — because one flag is 1 grapheme, 2 code points and 8 bytes, and all three numbers are true.
C3 AB out of U+00EB by hand; chapter 15 gave you the file descriptor and the rb door, and this chapter simply named what encoding= was bolting onto it — a TextIOWrapper is an encode and a decode with a buffer; chapter 16 pushed those bytes past the page cache to the platter, and never once asked what they meant; chapter 13 gave you try / except and the fact that UnicodeDecodeError is a ValueError underneath, so the net you already string catches it. Chapter 17 added barely any syntax — two methods and one keyword argument. What it added is a question you can now ask of every line that touches the outside world: which side of the wall am I on, and what key did I name?# robust_reader.py -- read text that somebody else's program wrote
from pathlib import Path
TRIES = ("utf-8-sig", "utf-8", "cp1252") # strictest first; latin-1 is NOT in here
def read_text_robust(path, tries=TRIES):
"""Return (text, report). Try each codec strictly, then fall back to lossy utf-8."""
raw = Path(path).read_bytes() # ch15's 'rb' door: bytes, no guessing
bom = raw[:3] == b"\xef\xbb\xbf"
for codec in tries:
try:
text = raw.decode(codec) # strict: it raises rather than lie
except UnicodeDecodeError:
continue
return text, {"codec": codec, "bom": bom, "bytes": len(raw),
"points": len(text), "lost": 0}
text = raw.decode("utf-8", errors="replace") # last resort, and it IS lossy
return text, {"codec": "utf-8+replace", "bom": bom, "bytes": len(raw),
"points": len(text), "lost": text.count("�")}
# --- four files, written by four different programs ---
LINE = "Zoë Müller — café"
here = Path(".")
(here / "a_utf8.txt").write_bytes(LINE.encode("utf-8"))
(here / "b_latin1.txt").write_bytes("Zoe Muller - cafe\n".encode("latin-1") + b"\xe9\xe8\xea")
(here / "c_bom.txt").write_bytes(b"\xef\xbb\xbf" + LINE.encode("utf-8"))
(here / "d_broken.txt").write_bytes(b"Zo\xc3\xab \x81\x8d truncated \xc3")
print(f"{'file':<14}{'codec':<15}{'bom':<7}{'bytes':>6}{'points':>7}{'lost':>6} first line")
for p in sorted(here.glob("?_*.txt")):
text, r = read_text_robust(p)
head = text.splitlines()[0] if text else ""
print(f"{p.name:<14}{r['codec']:<15}{str(r['bom']):<7}{r['bytes']:>6}"
f"{r['points']:>7}{r['lost']:>6} {head}")
p.unlink()
print()
print("utf-8-sig strips the BOM; plain utf-8 would leave U+FEFF at index 0.")
print("latin-1 stays out of TRIES: it decodes all 256 bytes, so it would always 'win'.")
print("a strict decode that SUCCEEDS is evidence, never proof -- cp1252 succeeds often.")
file codec bom bytes points lost first line a_utf8.txt utf-8-sig False 22 17 0 Zoë Müller — café b_latin1.txt cp1252 False 21 21 0 Zoe Muller - cafe c_bom.txt utf-8-sig True 25 17 0 Zoë Müller — café d_broken.txt utf-8+replace False 19 18 3 Zoë �� truncated � utf-8-sig strips the BOM; plain utf-8 would leave U+FEFF at index 0. latin-1 stays out of TRIES: it decodes all 256 bytes, so it would always 'win'. a strict decode that SUCCEEDS is evidence, never proof -- cp1252 succeeds often.
robust_reader.py and run it. This is the chapter assembled into one function you will genuinely reuse. Real data arrives from other people's programs, and it does not tell you its codec — a byte stream carries no record of how it was written. So we do the only honest thing: read the bytes once through chapter 15's rb door, then try codecs strictest first and let each one refuse. utf-8-sig leads because it eats a byte-order mark if there is one and behaves exactly like utf-8 if there is not. Then plain utf-8, then cp1252 for the Windows-era files. Notice what is missing from TRIES, because it is the whole design. latin-1 decodes all 256 byte values, so it never raises and would win every time, handing you confident garbage. Leaving it out is what keeps the lossy fallback reachable, and d_broken.txt proves the fallback fires: bytes 0x81 and 0x8d are undefined even in cp1252, so we land on utf-8+replace with lost: 3. That number is the point of the whole report. A zero means every byte was understood; a three means three were guessed at, and now you know to go and ask. Read the four rows and notice each column earning its place: c_bom.txt is three bytes longer than a_utf8.txt for the same seventeen characters, and the BOM column says why. And keep the last line honest with yourself — a strict decode that succeeds is evidence, not proof. cp1252 will happily accept most byte salad. When you need a real answer rather than a good guess, that is what charset-normalizer and chardet are for: they score candidates statistically instead of taking the first one that does not crash. Two things to try. Move latin-1 into TRIES and watch every file "succeed" and d_broken.txt come back as silent nonsense. Then add a utf-16 file and see it defeat the list entirely — its zero bytes are legal UTF-8, so the first codec wins with a string full of \x00.Text and bytes look alike but are different Python types: a str is a sequence of Unicode code points, while bytes is the raw form that lives on disk and travels the wire. UTF-8 is the bridge between them. These twelve programs keep the two side by side — encoding, decoding, counting, and deliberately breaking things — so the byte level stops being a mystery.