Encoding: ASCII, Unicode, Base64, and hex

Overview

Computers only ever store numbers — specifically, bytes (values from 0 to 255). Text, emoji, images, network packets, and program memory are all just bytes underneath. Encoding is the agreed-upon rule for turning meaningful data (like the letter A or a chunk of binary) into a sequence of bytes or printable characters, and back again.

The key idea is simple: encoding is a reversible way to represent data. It is not encryption. Encryption scrambles data with a secret key so that only someone holding the key can recover it. Encoding has no key at all — anyone who knows the format can reverse it instantly. Mixing up these two ideas is one of the most common beginner mistakes, and a frequent finding in real security reviews.

This lesson connects directly to the rest of Computer & OS Fundamentals. When you looked at environment variables, the values were stored as text; encoding is what decides how that text becomes bytes. In the next lesson, Compression and archives, you will see another reversible byte transformation — compression — that, like encoding, changes representation without adding secrecy.

Four encodings are worth recognising on sight:

Once you can tell these apart, a lot of "mysterious" strings in logs, tokens, and traffic stop being mysterious.

Why it matters

Encoding is everywhere in real software, and being fluent in it pays off daily.

It is the lingua franca of data interchange. Text-only channels — HTTP headers, JSON, email, URLs, log files — cannot carry arbitrary bytes safely. Base64 and hex let binary data (keys, images, signatures) travel through those channels intact. If you have ever seen a data:image/png;base64,iVBORw0... URL or an Authorization: Basic ... header, you have seen encoding doing its job.

It shows up constantly in security work.

• Analysts and pentesters decode Base64 and hex inside tokens, cookies, network captures, and config files all the time. Recognising the format is the first step to understanding what a system is actually doing.
• Encoding tricks — such as UTF-8 overlong forms or Unicode normalisation — can slip malicious input past naive filters that only check the "obvious" representation. Defenders need to decode first, then validate.
• Confusing encoding with encryption is itself a finding. "The password is Base64-encoded, so it is safe" is wrong: anyone can reverse it in one step. Calling that out correctly is part of a competent review.

It prevents data corruption. Choosing the wrong encoding (treating UTF-8 bytes as ASCII, or forgetting Base64 padding) produces garbled text and broken files — the classic "mojibake" you have seen when an accented name turns into é. Understanding encoding is how you diagnose and fix that.

Core concepts

Each encoding solves a slightly different problem. We will define each one, see how it works internally, and note when not to use it.

1. Encoding vs. encryption (the foundation)

Definition. Encoding transforms data into a specific format using a public, fixed rule. Encryption transforms data using an algorithm plus a secret key so it cannot be reversed without that key.

Why it matters internally. Encoding is a pure lookup/arithmetic operation — there is nothing secret to hide. Encryption mixes in key material so the output looks random and is infeasible to reverse without the key.

ENCODING  (no secret):   data --[public rule]--> bytes --[public rule]--> data
ENCRYPTION (secret key): data --[algo + KEY ]--> ciphertext --[algo + KEY]--> data
                                        ^ remove the key and you are stuck

When NOT to confuse them. Never store a secret "protected" only by encoding. Base64 of a password protects nothing.

Pitfall. Seeing random-looking characters and assuming "it's encrypted." Random-looking is not the same as secret.

Knowledge check. A teammate says, "Our API keys are safe in the config file because they're Base64-encoded." In your own words, why is this claim wrong, and what would actually protect them?

2. ASCII

Definition. ASCII assigns the numbers 0–127 to a basic set of characters: English letters, digits, punctuation, space, and some control codes.

Plain language. It is a tiny lookup table. The letter A is the number 65; a is 97; the digit 0 is 48; space is 32.

How it works internally. Each character fits in a single byte, and the top bit (the 128 place) is always 0 for true ASCII. That is why ASCII text is also valid UTF-8.

Structure (a slice of the table).

Dec  Char     Dec  Char     Dec  Char
 32  (space)   65  A         97  a
 48  0         66  B         98  b
 49  1         90  Z        122  z
 10  \n (LF)   13  \r (CR)   00  NUL

When to use / not to use. Great for simple English text and protocol keywords. Do not use plain ASCII for anything beyond the 0–127 range (accents, other alphabets, emoji) — it cannot represent them.

Pitfall. Assuming '9' - '0' == 9 works for characters but forgetting that '9' itself is the byte 57, not the number 9. The subtraction works because the digits are contiguous in the table.

Knowledge check (predict the output). If A is 65 and the letters are contiguous, what is the ASCII code for D?

3. Unicode and UTF-8

Definition. Unicode is a giant catalogue that assigns a unique number (a "code point") to every character in every writing system, plus emoji and symbols. UTF-8 is the most common way to encode those code points as bytes.

Plain language. Unicode says "é is code point U+00E9." UTF-8 says "here is how to write U+00E9 as actual bytes."

How it works internally. UTF-8 is variable-length: 1 byte for code points 0–127 (identical to ASCII), and 2–4 bytes for everything else. The leading bits of the first byte announce how many bytes follow.

Code point range        UTF-8 bytes (binary pattern)
U+0000 – U+007F         0xxxxxxx                    (1 byte, == ASCII)
U+0080 – U+07FF         110xxxxx 10xxxxxx           (2 bytes)
U+0800 – U+FFFF         1110xxxx 10xxxxxx 10xxxxxx  (3 bytes)

Example: 'é' = U+00E9 -> bytes 0xC3 0xA9 (two bytes)

When to use / not to use. UTF-8 is the default for the web, files, and APIs — use it. Avoid assuming "one character = one byte"; that breaks the moment a non-ASCII character appears.

Pitfall (and a security angle). Different byte sequences can normalise to the same character, and some systems historically accepted "overlong" encodings. A filter that blocks the literal byte for / might miss an alternate encoding. The defensive rule: decode and normalise input to one canonical form, then validate.

Knowledge check. Why is plain ASCII text automatically valid UTF-8, but a UTF-8 file containing é is not valid ASCII?

4. Hexadecimal (hex)

Definition. Hex is base-16. Its digits are 0–9 then a–f (10–15). Two hex digits represent exactly one byte (0–255).

Plain language. It is a compact, human-readable way to write raw bytes. The byte 255 is FF; the byte 0 is 00; the byte 65 (A) is 41.

How it works internally. A byte has 8 bits = two groups of 4 bits ("nibbles"). Each nibble (0–15) maps to one hex digit, so any byte is always two hex characters.

Byte 65 = binary 0100 0001
            high nibble ^   ^ low nibble
            0100 = 4        0001 = 1
  -> hex "41"

Hex dump layout. A hex dump shows the byte offset, the bytes as hex, and the same bytes as printable ASCII (non-printable bytes shown as .):

Offset    Hex bytes                                         ASCII
00000000  48 65 6c 6c 6f 2c 20 77  6f 72 6c 64 21 0a        Hello, world!.

When to use / not to use. Use hex to display binary precisely (hashes, dumps, packet bytes). It is verbose — do not use it as a transport format when Base64 would be ~33% smaller.

Pitfall. Confusing the hex text "41" (two characters) with the byte 0x41 (one byte). Decoding hex always halves the length.

Knowledge check (find the bug). A log shows the SHA-256 of a file as a 63-character hex string. Why should that immediately look suspicious?

5. Base64

Definition. Base64 encodes arbitrary bytes using 64 printable characters (A–Z, a–z, 0–9, +, /), turning every 3 input bytes into 4 output characters, with = used as padding.

Plain language. It is the standard way to stuff binary data into a text-only slot. dGVzdA== decodes instantly to test.

How it works internally. Take 3 bytes = 24 bits. Split those 24 bits into four 6-bit groups. Each 6-bit group (0–63) indexes the 64-character alphabet.

Input bytes:   T (0x54)     e (0x65)     s (0x73)
Bits:          01010100     01100101     01110011
Regroup by 6:  010101  000110  010101  110011
Values:           21      6       21      51
Alphabet:         V       G        V       z   ->  "VGVz"

Padding. If the input length is not a multiple of 3, Base64 pads the final group with = so the output length stays a multiple of 4: 1 leftover byte -> XX==, 2 leftover bytes -> XXX=.

When to use / not to use. Use it to carry binary through text channels (HTTP Basic auth, JWT segments, email MIME attachments, data URIs). Do not use it for compression (it makes data ~33% larger) or for secrecy (it is trivially reversible).

Pitfall. Dropping or mangling the = padding, or confusing standard Base64 (+ /) with URL-safe Base64 (- _). A wrong alphabet or missing padding causes decode failures.

Knowledge check. Roughly how many Base64 characters does a 30-byte input produce, and why is it not exactly 30?

Syntax notes

There is no single "syntax" for encoding the way there is for a programming construct, but there are standard command-line tools and recognisable shapes. Here is an annotated tour you can run on any Linux/macOS shell:

# ASCII code of a character (prints the decimal value)
printf 'A' | od -An -tu1        # -> 65   (od = octal dump; -tu1 = unsigned 1-byte ints)

# Hex of some text
printf 'Hi' | xxd               # -> 00000000: 4869   Hi   (offset, hex, ASCII)

# Encode text to Base64
printf 'test' | base64          # -> dGVzdA==   (note the == padding)

# Decode Base64 back to bytes
printf 'dGVzdA==' | base64 -d   # -> test

Recognising encodings by their shape:

Hex      : only 0-9 a-f, length is even, often lowercase   e.g. 48656c6c6f
Base64   : A-Z a-z 0-9 + /, length multiple of 4, may end = e.g. SGVsbG8=
UTF-8    : plain text; non-ASCII chars take >1 byte on disk
ASCII    : plain English text, every byte <= 127

A quick mental test: Is it only 0-9a-f and even length? probably hex. Does it end in = or == and use + /? probably Base64.

Lesson

Encoding represents data in a particular format. It is not encryption.

There is no secret key, and anyone can reverse it. Confusing the two is a classic beginner mistake — and a common audit finding.

ASCII

ASCII maps the basic English characters to the numbers 0–127.

• A = 65
• a = 97
• space = 32

Each character takes one byte.

Unicode / UTF-8

Unicode covers every character in every language.

UTF-8 is a way to encode Unicode using 1 to 4 bytes per character. It stays ASCII-compatible for the first 128 code points, which is why it is the web's default encoding.

Encoding tricks (such as overlong or normalised forms) can bypass naive input filters. This is a real web-security topic.

Hex

Hex is base-16. Its digits are 0-9 and a-f.

Each byte is written as two hex digits — for example, 0xFF = 255.

Hex is used wherever binary data must be shown as text: hashes, memory dumps, and packet bytes. A hex dump shows raw bytes as hex alongside their printable ASCII.

Base64

Base64 encodes arbitrary bytes using 64 printable characters (A-Za-z0-9+/). It turns every 3 bytes into 4 characters, sometimes adding = as padding.

It is used to carry binary data through text-only channels:

• HTTP Basic authentication
• JWTs
• Email attachments
• Data URIs

Base64 is not encryption. For example, dGVzdA== decodes instantly to test. Spotting Base64 in traffic and decoding it is routine recon work.

Code examples

Below is a self-contained worked example you can reproduce in a shell. It encodes the same short message four ways so you can see the representations side by side, then decodes them back to prove encoding is reversible with no key.

#!/usr/bin/env bash
# encoding_demo.sh — show ASCII, UTF-8, hex, and Base64 for one message.
set -euo pipefail               # stop on errors / undefined vars (robust scripting)

MSG='Hi!'                       # 3 ASCII bytes: H=72 i=105 !=33

echo "Original text : $MSG"

# 1) ASCII codes (decimal value of each byte)
echo -n "ASCII codes  : "
printf '%s' "$MSG" | od -An -tu1 | tr -s ' '

# 2) Hex (two hex digits per byte)
echo -n "Hex          : "
printf '%s' "$MSG" | xxd -p     # -p = plain hex, no offsets/columns

# 3) Base64 (3 bytes -> 4 chars; here 3 bytes -> exactly 'SGkh')
ENC=$(printf '%s' "$MSG" | base64)
echo "Base64       : $ENC"

# 4) Decode the Base64 back to the original (reversible, no key needed)
DEC=$(printf '%s' "$ENC" | base64 -d)
echo "Decoded back : $DEC"

# 5) A non-ASCII character to show UTF-8 multi-byte behaviour
printf 'UTF-8 of e-acute (é): '
printf '\xc3\xa9' | xxd -p       # é = U+00E9 = bytes c3 a9 in UTF-8

What it does. It prints the message Hi! as decimal ASCII codes, as plain hex, and as Base64, then decodes the Base64 back to Hi! to demonstrate reversibility. Finally it shows that the single character é occupies two bytes (c3 a9) in UTF-8.

Expected output.

Original text : Hi!
ASCII codes  :  72 105 33
Hex          : 486921
Base64       : SGkh
Decoded back : Hi!
UTF-8 of e-acute (é): c3a9

Why these values. H=72=0x48, i=105=0x69, !=33=0x21, which concatenate to the hex 486921. Because Hi! is exactly 3 bytes, Base64 produces exactly 4 characters (SGkh) with no = padding. The decode step recovers the original with nothing but the public Base64 rule.

Key edge cases. If MSG were Hi (2 bytes), Base64 would emit SGk= (one =). If it were H (1 byte), you would get SA== (two =). A non-ASCII message changes the byte count: é alone is 2 bytes, so its Base64 is w6k=, not a single 4-char group of one byte.

Line by line

Here is a step-by-step trace of the demo, focusing on how the bytes flow through each representation.

1. set -euo pipefail — defensive scripting: exit on any command failure, on use of an unset variable, and on a failure anywhere in a pipeline. This is unrelated to encoding but is good habit.
2. MSG='Hi!' — stores three characters. In memory these are the bytes 72 105 33.
3. od -An -tu1 — reads the bytes and prints each as an unsigned 1-byte integer. -An drops the address column so you see only the values. Output: 72 105 33.
4. xxd -p — prints the same bytes as plain hex, two digits each, concatenated: 48 69 21 -> 486921.
5. base64 — groups the 3 bytes into 24 bits, splits into four 6-bit values, and maps each to the alphabet (see trace table below), giving SGkh.
6. base64 -d — reverses step 5 using the same public table, reconstructing 72 105 33 -> Hi!. No key is involved; that is the whole point.
7. printf '\xc3\xa9' | xxd -p — writes the two raw UTF-8 bytes for é and shows them as c3a9, demonstrating that one visible character can be more than one byte.

Base64 trace for Hi!:

Byte    H=0x48      i=0x69      !=0x21
Bits    01001000    01101001    00100001
24 bits 010010 000110 100100 100001
Value     18     6      36      33
Char      S      G       k       h     -> "SGkh"

Reading the table top to bottom shows exactly how 3 bytes become 4 characters: regroup 8-bit bytes into 6-bit chunks, look up each chunk, done. Because there were no leftover bytes, there is no = padding.

Common mistakes

Mistake 1 — "It's Base64, so it's secure."

Wrong: storing or transmitting a secret like cGFzc3dvcmQxMjM= and treating it as protected.

Why it's wrong: anyone can run base64 -d and get password123 back. There is no key, so there is no secrecy.

Corrected: protect secrets with real cryptography or a secrets manager; if you must keep them in config, encrypt them with a key you control and store passwords hashed (e.g. with a slow password hash), never reversibly encoded.

How to recognise it: if you can decode it without any key, it is not protecting anything.

Mistake 2 — Treating UTF-8 bytes as ASCII (assuming 1 char = 1 byte).

Wrong: len("café") logic that counts bytes and reports 5 while the user sees 4 characters, or truncating a string at a fixed byte offset and slicing a multi-byte character in half.

Why it's wrong: é is two bytes in UTF-8; byte length and character count are different things.

Corrected: use Unicode-aware length and slicing functions, and treat text as UTF-8 end to end. When truncating, cut on character boundaries.

How to recognise it: output shows é or ? where an accented or non-Latin character should be — classic mojibake.

Mistake 3 — Validating input before decoding it.

Wrong: a filter rejects the literal string ../ but lets an alternate-encoded form through, which is decoded later into ../.

Why it's wrong: the dangerous value only appears after decoding, so a pre-decode check misses it.

Corrected: canonicalise first — decode and normalise to one form — then validate against an allow-list.

How to recognise it: security tests that pass with plain input but fail with encoded equivalents.

Mistake 4 — Mismatched Base64 alphabet or dropped padding.

Wrong: decoding a URL-safe token (which uses - and _) with a standard decoder, or stripping the trailing =.

Why it's wrong: the decoder cannot map -/_ (standard expects +//), and missing padding can make the length invalid.

Corrected: use the URL-safe decoder for URL-safe data; re-add padding if your library requires it.

How to recognise it: "invalid character" or "incorrect padding" decode errors, especially on JWT or query-string values.

Debugging tips

When a decode fails or text looks garbled, work through these:

• "Invalid base64" / "incorrect padding". Check the length is a multiple of 4 (add = if your tool needs it), and check the alphabet: does the string contain - or _? That is URL-safe Base64 — use the URL-safe decoder. Whitespace or newlines inside the string can also break strict decoders; strip them first.

• Hex won't decode. Confirm the length is even (one byte = two hex digits) and that every character is 0-9a-fA-F. An odd length or a stray 0x prefix or spaces are common culprits.

• Garbled accents / mojibake (é, ’, ?). This is an encoding mismatch: data written as UTF-8 is being read as something else (or vice versa). Verify the byte sequence with xxd; é should be c3 a9. Make sure files, database connections, and HTTP headers all declare and use UTF-8.

• Wrong ASCII arithmetic. If converting a digit character to a number gives a value like 57 instead of 9, you forgot to subtract '0' (48). Remember the character '9' is the byte 57.

• "It decoded but it's still gibberish." You may have decoded the right format to the wrong thing — e.g. Base64 that decodes to more Base64, or to compressed/binary data. Inspect the decoded bytes with a hex dump before assuming the format.

Questions to ask when it doesn't work:

1. What encoding did the producer use, and what is the consumer assuming?
2. Is the alphabet standard or URL-safe? Is padding present and correct?
3. Is the length plausible (even for hex, multiple of 4 for Base64)?
4. Have I looked at the raw bytes (xxd / od) instead of trusting the rendered text?

Memory safety

This is a concept lesson, so the focus is on robustness and validation rather than C memory bugs — but the habits transfer directly to any code that handles encoded data.

Validate before you allocate and before you trust.

• Length math is predictable — use it. Decoded hex is exactly half the input length; decoded Base64 is roughly 3/4. When you size a buffer for the decoded output, compute it from the input length and check for overflow before allocating. A maliciously long input should be rejected, not silently truncated.
• Never assume the input is well-formed. Encoded data often comes from untrusted sources (URLs, headers, uploaded files). Reject invalid characters, odd hex lengths, and bad Base64 padding instead of guessing.
• Decode-then-validate, in that order. As noted in the mistakes section, the dangerous value may only appear after decoding. Canonicalise to one form, then check against an allow-list. Validating the raw encoded string is not enough.
• One character is not one byte. When working with UTF-8 in C or any language, do not index or truncate text by byte offset expecting character boundaries — you can split a multi-byte character and produce invalid UTF-8. Treat the buffer as bytes and only split on known boundaries.
• Do not log secrets. When debugging an encoded token or credential, log that decoding succeeded/failed and maybe the length — never the decoded secret itself. Encoding gives zero confidentiality, so a decoded token in a log is a leaked token.

Real-world uses

Concrete real-world uses:

• HTTP Basic authentication sends username:password Base64-encoded in an Authorization: Basic ... header. (This is exactly why Basic auth must run over HTTPS — the encoding provides no protection on its own.)
• JWTs are three Base64url-encoded segments (header.payload.signature) joined by dots; the first two decode to readable JSON, which is why you should never put secrets in a JWT payload.
• Email attachments use Base64 (MIME) to carry binary files through text-only mail transport.
• Data URIs embed images and fonts directly in HTML/CSS as data:...;base64,....
• Hashes and fingerprints (SHA-256, SSH key fingerprints, certificate thumbprints) are displayed as hex or Base64 so humans and tools can compare them.
• Hex dumps are a daily tool in debugging, reverse engineering, file-format work, and network packet analysis.

Professional best-practice habits.

Beginner rules:

• State the encoding explicitly everywhere — file headers, HTTP Content-Type; charset=utf-8, database/connection settings. Default to UTF-8.
• Never call encoding "encryption" or treat encoded data as confidential.
• Use battle-tested library functions for encode/decode; do not hand-roll Base64 in production.

Advanced rules:

• Canonicalise then validate all untrusted input; normalise Unicode to a single form (e.g. NFC) before comparisons or security checks.
• Choose the right transport encoding: Base64url for URLs/JWTs, standard Base64 for MIME, hex when humans must read individual bytes.
• Be deliberate about size and performance: Base64 inflates data ~33%; for large binary payloads prefer a binary channel when one is available.
• In logs and error messages, record outcomes and lengths, never the decoded secret.

Practice tasks

Beginner 1 — Encode by hand. Objective: encode the text Cat to ASCII codes and to hex without a computer, then verify with a tool. Requirements: write the three ASCII decimal codes, then the hex string. Verify with printf 'Cat' | xxd -p. Expected output: ASCII 67 97 116; hex 436174. Hint: C=67, then count up the alphabet for lowercase. Concepts: ASCII table, byte-to-hex (two digits per byte).

Beginner 2 — Spot the format. Objective: classify each string as hex, Base64, or plain ASCII, and justify your choice. Strings: 48656c6c6f, SGVsbG8=, Hello. Requirements: for each, state the format and the clue you used (character set, even length, trailing =, length multiple of 4). Then decode the first two and confirm they all represent Hello. Hint: hex is only 0-9a-f and even length; Base64 may end in =. Concepts: recognising encodings by shape.

Intermediate 1 — Base64 padding. Objective: predict the Base64 padding for inputs of different lengths. Requirements: without running it first, predict the Base64 output (and number of =) for A, AB, and ABC. Then verify each with printf '%s' AB | base64. Expected pattern: 1 byte -> two =; 2 bytes -> one =; 3 bytes -> no =. Constraints: explain why using the 3-bytes-to-4-chars rule. Concepts: Base64 grouping and padding.

Intermediate 2 — Make the audit finding. Objective: write a two-sentence finding for a config file containing db_password: cGFzc3dvcmQ=. Requirements: decode the value, state clearly that Base64 is encoding (not encryption), explain the risk, and recommend a remediation (secrets manager / encryption-at-rest / hashing where applicable). Constraints: do not print the decoded password in any shared report; refer to it as a secret. Concepts: encoding vs. encryption, defensive handling of secrets.

Challenge — Round-trip checker. Objective: design (in pseudocode or your language of choice) a small tool that takes a string and a claimed encoding (hex or base64), decodes it, re-encodes it, and reports whether the round-trip matches the original. Requirements: validate the input first (correct alphabet; even length for hex; length multiple of 4 and valid padding for Base64) and return a clear error instead of crashing on bad input. Handle both standard and URL-safe Base64. Input/output example: input ("SGVsbG8=", "base64") -> decodes to Hello, re-encodes to SGVsbG8=, reports MATCH. Constraints: never log the decoded bytes if they look like a secret; compute the decoded buffer size from the input length and reject implausibly large inputs. Concepts: validation-before-decode, length math, alphabet/padding handling, robust error reporting.

Summary

• Encoding is representation, not protection. It is a public, reversible rule with no key — fundamentally different from encryption. Calling Base64 "secure" is wrong and is a real audit finding.
• ASCII maps 0–127 to basic English characters, one byte each (A=65). UTF-8 encodes all of Unicode in 1–4 bytes and is ASCII-compatible — so one character is not always one byte.
• Hex writes each byte as two digits (A=41); decoding halves the length, and the length is always even. Base64 turns every 3 bytes into 4 characters from A-Za-z0-9+/, padding with =; decoding gives back ~3/4 the length.
• Most important commands to remember: xxd -p (hex), base64 / base64 -d (encode/decode), od -tu1 (byte values).
• Common mistakes: treating encoded data as secret, assuming 1 char = 1 byte, validating before decoding, and mismatching the Base64 alphabet or padding.
• What to remember: recognise the format by its shape, decode-then-validate untrusted input, default to UTF-8, compute buffer sizes from input length, and never log decoded secrets.

Practice with these exercises

• Base64 encoded length
• Valid base64 character?
• Print a hex dump line
• Bounded base64 decoder