Text is bytes too: encodings

How letters become numbers

Every file you read, API you call, and database row you store crosses a text-to-bytes boundary. When a customer named José shows up as José in production, or a CSV from a client refuses to parse, this lesson is the one that fixes it.

RAM stores bits, so the letter A must secretly be a number. The oldest agreement is ASCII, where A is 65, B is 66, and a is 97, covering English letters, digits, and punctuation in the numbers 0 to 127. One character fit in one byte and life was simple.

But the world writes in thousands of scripts. Unicode is the modern agreement, a giant table giving every character on Earth a number called a code point. A is still 65, é is 233, and 🐍 is 128013.

Python gives both directions, with ord() turning a character into its number and chr() turning a number back into a character. Note that Unicode assigns the numbers and says nothing about how to store them, which is the job of an encoding later in this lesson.

Characters, numbers, and bytes

ord and chr convert between the two, and .encode produces raw bytes.

print(ord("A"))
print(ord("B"))
print(chr(67))
data = "Hi!".encode("utf-8")
print(data)
print(len(data))

Output

65
66
C
b'Hi!'
3

The b'...' prefix means bytes rather than text. For plain English letters UTF-8 uses exactly one byte each, which is why three characters produce three bytes and the bytes still look like the original letters.

That readability is deliberate and is the reason UTF-8 won. Any file that was valid ASCII is already valid UTF-8 with identical bytes, so decades of existing files needed no conversion.

UTF-8, fitting big numbers into bytes

A code point such as 128013 does not fit in one byte, since bytes max out at 255 as seen in lesson 2-1. An encoding is the rule for packing code points into bytes, and the winner, used by about 98% of the web, is UTF-8.

So in Python len(string) counts characters while len(string.encode("utf-8")) counts bytes, and the two differ as soon as text leaves plain English.

Files, networks, and disks always carry bytes, never characters, so the conversion happens at every boundary whether you write it or not. A database column sized in bytes will therefore hold fewer characters of French than of English, which is a real source of truncation bugs.

When character count and byte count disagree

One accented word and one emoji, measured both ways.

word = "café"
print(len(word))
encoded = word.encode("utf-8")
print(encoded)
print(len(encoded))
print(len("🐍".encode("utf-8")))

Output

4
b'caf\xc3\xa9'
5
4

Four characters, five bytes. The first three letters cost one byte each and é costs two, shown as \xc3\xa9 in the hex notation from lesson 2-1, which is the bytes 0xc3 and 0xa9, or 195 and 169.

The single emoji costs four bytes on its own. Any code that budgets storage or truncates text by counting characters is therefore making an assumption that fails on the first non-English input it meets.

Mojibake, decoding with the wrong rule

Bytes do not know what encoding made them. Decoding UTF-8 bytes with a different rule produces garbage famous enough to have a name: mojibake.

The two bytes of é, read one at a time by the old latin-1 rule, become é. Nothing was corrupted in transit, and the bytes are byte-for-byte what was written.

If you have ever seen ’ where an apostrophe should be, or café on a menu website, you have watched this exact bug. The apostrophe case is the same story with a three-byte character.

The fix is always the same. Know your encoding, and in modern code use UTF-8 everywhere, declaring it explicitly when reading and writing files rather than relying on a platform default.

Mojibake is recoverable as long as nothing re-encoded the mangled text. Once the garbage characters are themselves saved as UTF-8, the original bytes are gone and the damage is permanent.

Producing mojibake on purpose

The same bytes decoded twice, once correctly and once with the wrong rule.

raw = "café".encode("utf-8")
print(raw.decode("utf-8"))
print(raw.decode("latin-1"))

Output

café
café

raw is identical in both calls, so nothing about the data changed. Only the decoding rule differed, and latin-1 treats each byte as one character, turning the two bytes of é into two separate letters.

This is also why mojibake never raises an error in latin-1. Every one of the 256 byte values is a valid latin-1 character, so the wrong rule always succeeds and always lies, which is worse than failing.

Measuring a word twice

Characters and UTF-8 bytes for a word with one accent.

word = "naïve"
print(len(word))
print(len(word.encode("utf-8")))

Output

5
6

len(word) counts characters and gives 5. Encoding first and then measuring gives 6, because the ï costs two bytes while the other four letters cost one each.

The order of the two operations is the whole point. len(word.encode("utf-8")) measures the thing that will actually be written to a file or sent over a network, which is the number that storage limits and network payloads care about.

Diagnosing café from the bytes

A file holding the five bytes b'caf\xc3\xa9' that a program reads as latin-1 and shows as café has been written as UTF-8 and decoded with a different rule. That is textbook mojibake.

The bytes are perfectly fine. UTF-8 stores é as two bytes, and latin-1 wrongly reads those as two separate characters, Ã and ©.

Decoding with the same encoding used to encode renders it as café again, and no repair of the file itself is needed. The bug is in the reader, which is where to look first whenever text arrives mangled.