Mojibake, Base64 and percent-encoding: how text really travels

Every so often, text arrives broken: an email where "café" reads "café", a filename full of %20, an API that wants your image "as Base64". These look like unrelated annoyances. They are all the same subject: computers do not store or transmit characters — they handle bytes — and everything interesting about text is in the translation between the two.

Bytes versus characters

A byte is a number from 0 to 255. A character — A, é, 漢, an emoji — is an abstract thing that needs a convention to become bytes. That convention is a character encoding, and the crucial fact is that the bytes alone do not tell you which encoding was used. The byte 0xE9 is é in Latin-1, part of a multi-byte sequence in UTF-8, and something else again in a Cyrillic encoding. Text is only meaningful as bytes plus an agreed encoding; lose track of the encoding and the bytes are just numbers.

For decades this was chaos: ASCII covered 128 characters (English letters, digits, punctuation) in one byte each, and dozens of incompatible regional encodings fought over the remaining 128 byte values. Unicode fixed the character inventory — one number, a code point, for every character in every script, written like U+00E9 for é — and UTF-8 became the standard way to turn code points into bytes.

UTF-8: variable width, by design

UTF-8 uses a different number of bytes for different characters:

• 1 byte for ASCII (U+0000–U+007F): A is 0x41, exactly as it was in 1968.
• 2 bytes for most Latin accents, Greek, Cyrillic, Hebrew, Arabic: é is 0xC3 0xA9.
• 3 bytes for most of the rest, including Chinese, Japanese and Korean: 漢 is three bytes.
• 4 bytes for code points beyond U+FFFF — which includes virtually all emoji: 😀 is 0xF0 0x9F 0x98 0x80.

The design is cleverer than "variable width" suggests. Every ASCII file is automatically valid UTF-8, unchanged — that backwards compatibility is why UTF-8 conquered the web (it is the declared encoding of well over 98% of web pages). And the byte patterns are self-synchronising: a continuation byte can never be mistaken for the start of a character, so software can find character boundaries mid-stream.

One practical consequence: "length" is ambiguous. A single emoji is 1 visible symbol, 1 code point, 4 UTF-8 bytes — and 2 units in JavaScript's or Java's internal UTF-16 strings. When a database column, SMS gateway or API limit counts "characters", it may be counting any of these. If a length limit ever behaves strangely with accents or emoji, this is why.

Where mojibake comes from

Mojibake (from Japanese, "character transformation") is what you see when bytes written in one encoding are read using another. The classic case: é is stored as UTF-8, two bytes 0xC3 0xA9. A program that wrongly assumes Latin-1 or Windows-1252 reads those two bytes as two separate characters: Ã (0xC3) and © (0xA9). Hence "café". Each accented character doubles into a distinctive pair, so a page of UTF-8 misread as Latin-1 is instantly recognisable by the rash of Ã characters.

The reverse error — Latin-1 bytes read as UTF-8 — produces the replacement character �, because lone bytes like 0xE9 are invalid UTF-8 sequences. And when text survives two wrong round-trips, you get compounded garbage like é turning into é.

The important point is that mojibake is not corruption. The bytes are intact; only the interpretation is wrong. The fix is never to edit the text but to correct the declared encoding — the HTTP Content-Type charset, the HTML <meta charset="utf-8">, the database connection encoding, or the encoding a file is opened with. In 2026 the right answer is almost always: use UTF-8 everywhere, and say so explicitly at every boundary.

Base64: binary-safe transport, not secrecy

Many channels were designed for text and cannot carry arbitrary bytes safely: email's underlying standards, JSON strings, XML, URLs. Base64 exists for exactly this problem. It takes any bytes and re-expresses them using 64 safe characters — A–Z, a–z, 0–9, + and /, with = as padding. Three bytes (24 bits) become four characters of 6 bits each; that is the whole trick.

The arithmetic gives the well-known cost: output is 4/3 the size of the input, roughly 33% overhead (plus a little padding). This is why embedding images in HTML or CSS as data: URIs, or attaching files to email (which uses Base64 under MIME), inflates them by a third — reasonable for small icons, wasteful for photographs.

Two things must be said plainly, because they are misunderstood constantly:

• Base64 is not encryption. It is a reversible re-spelling of bytes with no key and no secret; decoding it is one function call available in every language and every browser console. Credentials, tokens or personal data "protected" by Base64 are protected by nothing. HTTP Basic authentication Base64-encodes username:password purely to make it header-safe — which is exactly why it is only acceptable over HTTPS.
• Base64 is not compression. It makes data larger, always.

A variant worth knowing: standard Base64's + and / clash with URL syntax, so Base64url substitutes - and _. That is the alphabet used in JWTs and many API tokens — if a decoder rejects a token that looks like Base64, this substitution is usually why.

Percent-encoding: making bytes URL-safe

URLs have their own reserved characters: / separates path segments, ? starts the query, & separates parameters, # starts the fragment, and spaces are outright forbidden. To carry arbitrary text, URLs use percent-encoding: each unsafe byte becomes % followed by its two-digit hexadecimal value. A space is byte 0x20, hence %20. é is percent-encoded via its UTF-8 bytes, becoming %C3%A9 — percent-encoding sits on top of UTF-8, not beside it.

Two details cause most real-world bugs. First, in the query-string convention inherited from HTML forms, a space may also appear as + — so + in a query string means a space, and a genuine plus sign must be %2B. Second, encoding must be applied to values, not to whole URLs: encode a complete URL and you convert its structural / and ? into literal characters (%2F, %3F), breaking it. This is precisely the difference between JavaScript's encodeURIComponent (for a single value — almost always the one you want) and encodeURI (for a URL whose structure must survive). Double-encoding — where %20 becomes %2520 because the % itself got encoded again — is the visible symptom of applying encoding twice.

HTML entities: the last mile

HTML has its own three special characters: < and > delimit tags, and & starts an entity. Text destined for a web page must escape them — <, >, &, plus " inside attribute values. This is not cosmetic: rendering user-supplied text without escaping is the root cause of cross-site scripting (XSS), because <script> in a comment field becomes live code in every reader's browser.

Entities are only for HTML output. They do not belong in JSON, URLs or databases — store the real characters (UTF-8 handles all of them) and escape at the final rendering step for whichever destination the text is bound: entities for HTML, percent-encoding for URLs, backslash escapes for JSON.

That is the unifying rule of this whole subject. There is one text (Unicode), one sensible byte encoding (UTF-8), and a set of thin, mechanical, reversible transport encodings applied at boundaries: Base64 for byte-hostile channels, percent-encoding for URLs, entities for HTML. Every mangled string you will ever meet is one of these applied twice, applied to the wrong layer, or not applied at all — and once you can name which, the fix is usually one line.