HTTP/1.1 chunked transfer encoding

Overview

When an HTTP/1.1 server sends a response, the receiver needs to know where the body ends. The simplest way is the Content-Length header: "the body is exactly 1024 bytes." But sometimes the sender does not know the length in advance — for example, it is streaming a file as it is generated, tailing a live log, or proxying another stream. For those cases HTTP/1.1 defines chunked transfer encoding, selected with the header Transfer-Encoding: chunked.

With chunked encoding the body is broken into pieces called chunks. Each chunk announces its own size first, then carries that many bytes. The receiver reads a size, reads that many data bytes, and repeats until it sees a chunk of size zero, which means "the body is finished." Because each chunk is self-describing, the sender can keep producing data forever without ever committing to a total length.

This lesson builds directly on HTTP/1.1 protocol structure, where you saw how a request or response is split into a start line, header lines, and a body. Chunked encoding is one specific way that body can be framed. It also builds on Safe parsing — the defensive parser shape: a chunked body is attacker-controlled input arriving over a socket, so every length you read must be validated before you trust it. The single most important idea in this lesson is that the size field is data from the network, and lenient handling of it is a security bug, not just a correctness bug.

The shape of one chunk is:

HEXLEN CRLF DATA CRLF

• HEXLEN — the chunk's size, written in hexadecimal (e.g. 1a means 26 bytes).
• CRLF — the two-byte line ending \r\n (carriage return + line feed).
• DATA — exactly that many bytes of body content.
• A trailing CRLF follows the data of every chunk.

The whole body ends with a zero-length chunk: 0\r\n\r\n.

Why it matters

Why this matters

Chunked encoding is everywhere. Every conforming HTTP/1.1 server and client must support it, and it is the standard framing for streamed responses such as Server-Sent Events, live log tails, and large proxied downloads. If you write any code that speaks HTTP/1.1 directly — a client, a server, a proxy, a load balancer, a WAF — you will parse chunked bodies.

The smuggling risk

Web traffic rarely reaches its destination through a single program. It usually passes through at least two parsers: a front-end (a reverse proxy, CDN, or load balancer) and a back-end (the origin server). Both must agree on exactly where one request ends and the next begins.

If the two parsers disagree about the body length — one trusts Content-Length, the other trusts Transfer-Encoding, or one accepts a malformed chunk size the other rejects — an attacker can append bytes that the front-end treats as the end of the body but the back-end treats as the start of a second request. That hidden request is then "smuggled" past the front-end's security checks. This is HTTP request smuggling.

This is not academic. A family of widely exploited smuggling vulnerabilities from 2019 to 2021 came from exactly this kind of parser disagreement, and the root cause was almost always a parser that accepted a sloppy chunk size or a sloppy combination of length headers. A strict chunked parser is a security control.

Core concepts

1. The chunk

Definition. A chunk is one self-sized piece of the body, framed as HEXLEN CRLF DATA CRLF.

Plain language. The sender says "here come 4 bytes," sends 4\r\n, sends Wiki, then closes the chunk with \r\n. The receiver does the inverse: read the size line, read that many bytes, read the closing CRLF.

How it works internally. The size is parsed from hex into an integer n. The parser then copies exactly n bytes from the input into the output buffer, then consumes the two-byte CRLF that follows. The size line itself is not part of the body — only DATA is.

Structure.

  4 \r \n  W  i  k  i  \r \n   <- one chunk: size=4, data="Wiki"
  5 \r \n  p  e  d  i  a  \r \n  <- next chunk: size=5, data="pedia"
  0 \r \n \r \n              <- terminator: size=0 ends the body

When to use it. Use chunked framing when the total length is unknown at the moment you start sending (streaming, proxying, generated output).

When NOT to. If you already know the length, send Content-Length — it is simpler and lets the receiver pre-allocate. Never send both Content-Length and Transfer-Encoding: chunked.

Pitfall. Forgetting that the data itself can contain \r\n bytes. You must copy exactly n bytes based on the size; you must not scan the data for a line ending. The CRLF after the data is structural, not a delimiter you search for.

Knowledge check (predict the output): Given the body 3\r\nabc\r\n0\r\n\r\n, how many bytes of decoded content are there, and what are they?

2. The size line and strict hex parsing

Definition. The size line is the chunk's length in hexadecimal, optionally followed by a ; name=value chunk extension, then a CRLF.

Plain language. 1a means 26. ff means 255. The number tells you how many data bytes to read next.

How it works internally. A strict parser reads characters until the CRLF, then converts only 0-9 a-f A-F into a number. Everything else is a parse error.

Strict rules — reject all of these:

When NOT to be lenient. Never. strtol with a leading-zero, sign-accepting, or 0x-accepting configuration is the classic mistake. Hand-write the digit loop or constrain strtol and then re-validate.

Pitfall. Using strtol(line, NULL, 16) and ignoring the end pointer. That silently accepts 4junk, leading +, surrounding spaces, and 0x4. Always pass an endptr and confirm it lands exactly on the CRLF.

Knowledge check (find the bug): A parser does long n = strtol(line, NULL, 16); and then reads n bytes. What three malformed size lines slip through, and what is the worst-case consequence of one of them?

3. The terminator and trailers

Definition. The body ends with a zero-length chunk: 0\r\n followed by a final \r\n. Between those two CRLFs, optional trailer header fields may appear.

Plain language. A chunk of size 0 means "no more data." The blank line after it closes the message. Trailers are extra headers that arrive after the body, used for values you cannot compute until the body is finished (such as a checksum).

Structure.

0\r\n               <- zero-length chunk: end of data
Expires: ...\r\n   <- optional trailer field (rare)
\r\n               <- final blank line: end of message

When to use trailers. Almost never as a beginner. They are rare and many intermediaries drop them.

When NOT to. Do not blindly accept arbitrary trailer headers — treat them with the same suspicion as normal headers, and accept only an allow-list of trailer names you actually expect.

Pitfall. Stopping at the 0\r\n and forgetting the final \r\n (or the trailer section). The message is not complete until the final blank line is consumed; leaving it in the buffer corrupts the next request on a kept-alive connection.

4. The smuggling threat model (why strictness is a control)

Definition. HTTP request smuggling exploits a disagreement between two HTTP parsers about where a body ends.

Data flow:

 attacker --> [ front-end proxy ] --> [ back-end server ]
                trusts X                 trusts Y
            (e.g. Content-Length)   (e.g. Transfer-Encoding)

 If front-end and back-end pick different lengths,
 bytes the front-end calls "body" become a
 second, hidden request to the back-end.

Defenses (each is a concrete rule your parser enforces):

• Reject any message that carries both Content-Length and Transfer-Encoding.
• Reject any size line that is not clean hex + CRLF (rules in concept 2).
• Cap each chunk's size and cap the running total of decoded bytes.
• Require an exact CRLF after each chunk's data; do not tolerate bare LF.

Knowledge check (explain in your own words): Why does rejecting a size line that has a leading zero make a smuggling attack harder, even though 04 and 4 are numerically equal?

Syntax notes

Syntax and structure

Chunked encoding is defined in RFC 7230, section 4.1. The grammar, simplified, is:

chunked-body = *chunk last-chunk trailer-part CRLF
chunk        = chunk-size [ chunk-ext ] CRLF chunk-data CRLF
chunk-size   = 1*HEXDIG          ; hexadecimal, at least one digit
last-chunk   = 1*("0") [ chunk-ext ] CRLF
trailer-part = *( header-field CRLF )

A minimal annotated example of the bytes on the wire:

4\r\n        <- chunk-size = 4 (hex)
Wiki\r\n     <- 4 data bytes, then closing CRLF
5\r\n        <- chunk-size = 5
pedia\r\n   <- 5 data bytes, then closing CRLF
0\r\n       <- last-chunk (size 0)
\r\n        <- final CRLF (no trailers here)

Key C building blocks you will use: strtol/manual hex conversion for the size, memcpy to copy exactly n data bytes, and realloc to grow the output buffer. Always validate n before using it as a length.

Lesson

When a server cannot predict the Content-Length ahead of time, for example while streaming, HTTP/1.1 lets it use the header Transfer-Encoding: chunked.

Each chunk is:

1. A length written in hex.
2. A CRLF.
3. That many bytes of body.
4. A CRLF.

A zero-length chunk ends the body.

If your parser disagrees with another parser about how the body ends, you have a smuggling vulnerability.

Code examples

Here is a complete, self-contained C11 decoder that parses a chunked body held in memory and writes the decoded bytes into a freshly allocated buffer. It parses strictly and enforces both a per-chunk cap and a total cap.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stddef.h>
#include <ctype.h>

#define MAX_CHUNK   (256u * 1024u)   /* reject any single chunk over 256 KiB */
#define MAX_TOTAL   (4u * 1024u * 1024u) /* cap the whole decoded body at 4 MiB */

/* Parse a strict hex chunk-size from [p, end). On success store the value in
 * *out and return a pointer just past the size digits; on error return NULL. */
static const char *parse_size(const char *p, const char *end, size_t *out) {
    if (p >= end || !isxdigit((unsigned char)*p)) return NULL; /* need >=1 digit */
    size_t val = 0;
    const char *start = p;
    while (p < end && isxdigit((unsigned char)*p)) {
        int d = isdigit((unsigned char)*p) ? *p - '0'
              : (tolower((unsigned char)*p) - 'a' + 10);
        if (val > (MAX_CHUNK - (size_t)d) / 16) return NULL; /* overflow / over cap */
        val = val * 16 + (size_t)d;
        p++;
    }
    /* Reject a leading zero unless the value is exactly "0" (the terminator). */
    if (*start == '0' && (p - start) > 1) return NULL;
    *out = val;
    return p;
}

/* Expect a literal CRLF at p; return pointer past it, or NULL. */
static const char *expect_crlf(const char *p, const char *end) {
    if (end - p < 2 || p[0] != '\r' || p[1] != '\n') return NULL;
    return p + 2;
}

/* Decode a chunked body in [src, src+len). On success returns a malloc'd
 * buffer (caller frees) and sets *out_len; on error returns NULL. */
static char *decode_chunked(const char *src, size_t len, size_t *out_len) {
    const char *p = src, *end = src + len;
    char *out = NULL;
    size_t cap = 0, used = 0;

    for (;;) {
        size_t n;
        const char *q = parse_size(p, end, &n);
        if (!q) goto fail;                 /* bad size line */
        /* Skip an optional chunk extension up to the CRLF, strictly. */
        while (q < end && *q != '\r') q++;
        q = expect_crlf(q, end);
        if (!q) goto fail;
        if (n == 0) { p = q; break; }      /* terminator chunk */

        if (used > MAX_TOTAL - n) goto fail;        /* total cap */
        if ((size_t)(end - q) < n) goto fail;       /* not enough data present */

        if (used + n > cap) {                        /* grow output buffer */
            size_t ncap = (cap ? cap : 1024);
            while (ncap < used + n) ncap *= 2;
            char *tmp = realloc(out, ncap);
            if (!tmp) goto fail;
            out = tmp; cap = ncap;
        }
        memcpy(out + used, q, n);          /* copy exactly n data bytes */
        used += n;
        q += n;

        q = expect_crlf(q, end);           /* CRLF after the chunk data */
        if (!q) goto fail;
        p = q;
    }

    /* After the terminator: optional trailers then a final CRLF. We skip any
     * trailer lines and require the closing blank line. */
    while (p < end && !(end - p >= 2 && p[0] == '\r' && p[1] == '\n')) {
        while (p < end && *p != '\n') p++;  /* skip a trailer line */
        if (p < end) p++;
    }
    if (!expect_crlf(p, end)) goto fail;    /* final CRLF must be present */

    *out_len = used;
    /* used may be 0 (empty body); return a valid 0-length buffer. */
    if (!out) out = malloc(1);
    return out;

fail:
    free(out);
    return NULL;
}

int main(void) {
    const char body[] = "4\r\nWiki\r\n5\r\npedia\r\n0\r\n\r\n";
    size_t n = 0;
    char *decoded = decode_chunked(body, sizeof body - 1, &n);
    if (!decoded) {
        fprintf(stderr, "reject: malformed chunked body\n");
        return 1;
    }
    printf("decoded %zu bytes: %.*s\n", n, (int)n, decoded);
    free(decoded);
    return 0;
}

What it does. decode_chunked walks the input chunk by chunk. For each chunk it parses a strict hex size, skips any extension up to the CRLF, copies exactly n data bytes into a realloc-grown buffer, and verifies the closing CRLF. A size of 0 ends the loop; the trailer/final-CRLF handling closes the message. Every length is bounds-checked against the bytes actually present and against MAX_CHUNK/MAX_TOTAL before use.

Expected output:

decoded 9 bytes: Wikipedia

Edge cases. An empty body (0\r\n\r\n) decodes to 0 bytes. A truncated input (size says 5 but only 3 bytes remain) is rejected by the end - q < n check. A size like 0xFFFFFFFF... is rejected by the overflow guard inside parse_size.

Line by line

Walkthrough of the example

Input on the wire: 4\r\nWiki\r\n5\r\npedia\r\n0\r\n\r\n

Why the result is Wikipedia (9 bytes). The two data chunks contribute Wiki (4) and pedia (5); concatenated in the output buffer that is Wikipedia. The size lines, the CRLFs, and the terminator are structure and never enter the output.

What is in memory at step 7. out points at a heap buffer of capacity 1024 (first grow), used == 4 holds Wiki, and memcpy(out + 4, "pedia", 5) appends the next five bytes so the buffer now begins with Wikipedia. The cursor q advances past the copied bytes before the closing-CRLF check.

The overflow guard. Inside parse_size, before val = val * 16 + d, the check val > (MAX_CHUNK - d) / 16 rejects any size that would exceed MAX_CHUNK. This both caps the chunk and makes integer overflow of val impossible.

Common mistakes

Common mistakes

1. Trusting strtol without checking the end pointer

Wrong:

long n = strtol(line, NULL, 16);   /* accepts "+4", " 4", "0x4", "4junk" */
read_n_bytes(fd, dst, n);

This silently accepts a leading sign, surrounding whitespace, a 0x prefix, and trailing garbage — every one of which is a smuggling foothold. A negative result from a +/- sign passed as a length is catastrophic.

Corrected:

char *endp;
errno = 0;
unsigned long n = strtoul(line, &endp, 16);
if (errno || endp == line || *endp != '\r')  return -1; /* must stop on CR */
if (line[0] == '0' && endp - line > 1)        return -1; /* leading zero */
if (n > MAX_CHUNK)                            return -1; /* cap */

Recognize it: if your tests pass for 4 but you never tried 4, 04, or 4x, you have not tested strictness.

2. Scanning the data for \r\n instead of using the size

Wrong: treating the chunk data as a line and reading until the next CRLF. Body data legitimately contains \r\n bytes, so this truncates or desynchronizes the parser.

Corrected: copy exactly n bytes (the size), then expect the CRLF as a separate structural step.

3. Forgetting the final CRLF / trailers

Wrong: breaking out of the loop on size 0 and returning immediately, leaving \r\n (and any trailers) unconsumed in the connection buffer. On a keep-alive connection the next request now starts with stray bytes.

Corrected: after the terminator, skip trailer lines and consume the final blank line before declaring the message complete (as in the example).

4. No total cap

Wrong: realloc the output for every chunk with no ceiling. A sequence of chunks (or one huge declared size) drives the process to out-of-memory.

Corrected: check used > MAX_TOTAL - n before allocating.

Debugging tips

Debugging tips

Compiler errors

• implicit declaration of 'isxdigit' → include <ctype.h>.
• Sign-comparison or conversion warnings on lengths → compile with -Wall -Wextra and keep sizes as size_t, never int or long you forgot to bound.

Runtime errors

• A crash inside memcpy almost always means n was not bounds-checked against the bytes actually present (end - q < n). Add the check before the copy.
• Run under sanitizers: cc -g -fsanitize=address,undefined chunk.c && ./a.out. ASan will pinpoint an over-read; UBSan will catch size-arithmetic overflow if your guard is wrong.

Logic errors

• Output is shorter/longer than expected → you are off by the CRLFs. Confirm you consume the CRLF after the data and that the size line CRLF is separate.
• Parser accepts input it should reject → log the exact byte where parsing diverged and compare against the RFC 7230 grammar.

Crafting test input by hand

To feed a parser raw chunked bytes:

• printf '4\r\nWiki\r\n5\r\npedia\r\n0\r\n\r\n' piped into your program reproduces the example exactly.
• curl -T - --header 'Transfer-Encoding: chunked' http://localhost:8080/ sends a chunked request from the shell to a local test server.
• nc -l 8080 (netcat in listen mode) lets you type or paste chunk bytes by hand to a local listener.

Questions to ask when it doesn't work

• Did I parse the size as strict hex, and confirm the end pointer landed on CR?
• Did I copy exactly n bytes, not "until the next CRLF"?
• Did I check n against both the remaining input and my caps before using it?
• Did I consume the trailing CRLF after every chunk and the final blank line?

Memory safety

Memory safety and robustness

This is not a security-category lesson, but parsing untrusted network input is exactly where C memory-safety bugs become exploitable. Apply these for this topic:

• Bounds before use. Always verify end - cursor >= n before memcpy(..., n). The size came from the wire; treat it as hostile until checked.
• Integer overflow of the size. A size like 0xFFFFFFFFFFFFFFFF must be rejected at parse time. The guard val > (MAX_CHUNK - d) / 16 both caps the value and prevents val from wrapping. Never read n bytes from a size you have not bounded.
• Integer overflow of the running total. Check used > MAX_TOTAL - n (subtraction form) rather than used + n > MAX_TOTAL (which can itself overflow).
• Allocation failure. realloc can return NULL; assign to a temporary, check it, and only then update out/cap. Overwriting out directly leaks the old block on failure.
• Lifetimes and cleanup. On any parse error, free(out) before returning (the fail: label) so a malformed body cannot leak memory across many requests.
• Initialization. cap and used start at 0 and out at NULL so the first grow path works and free(NULL) is safe.
• OOM as a denial of service. Accepting an unbounded total decoded length lets one request exhaust process memory. The total cap is a robustness and availability control.

Real-world uses

Real-world uses

• Streaming responses. Server-Sent Events, live log tailing, and large generated downloads all use chunked encoding because the length is unknown when the response begins.
• Reverse proxies and CDNs. nginx, HAProxy, Envoy, and every CDN parse and re-emit chunked bodies; their strictness about chunk sizes is what stands between users and request smuggling.
• HTTP client libraries. curl/libcurl, Go's net/http, and Python's http.client all decode chunked bodies; understanding the framing helps you debug truncated or hung transfers.
• WAFs and API gateways. Inspecting a body for policy enforcement requires decoding the chunks first; a lenient decoder here is a bypass.

Professional best-practice habits

Beginner rules:

• Parse the size as strict hex and confirm the line ends in CRLF.
• Copy exactly n bytes; never search the data for a delimiter.
• Always cap per-chunk and total sizes.
• Free the output buffer on every error path.

Advanced habits:

• Reject messages that carry both Content-Length and Transfer-Encoding.
• Use an allow-list for trailer field names; drop unexpected trailers.
• Reject bare LF line endings; require exact CRLF to match peer parsers.
• Fuzz the parser (e.g. with libFuzzer + ASan/UBSan) against truncated, oversized, and malformed inputs.
• Make the decoder streaming/incremental for production so it does not buffer the entire body, while still enforcing the total cap.

Practice tasks

Practice tasks

Beginner 1 — Decode a fixed fixture

Objective. Decode a known three-chunk body into one flat buffer. Requirements. Given the literal 3\r\nabc\r\n3\r\ndef\r\n0\r\n\r\n, decode and print the result. Expected output. abcdef (6 bytes). Constraints. Use memcpy for the data; do not scan for CRLF inside data. Hint. Reuse the size-line → data → CRLF loop shape from the lesson. Concepts. Chunk format, strict size parsing.

Beginner 2 — Detect the last chunk

Objective. Write int is_last_chunk(const char *hexsize) returning 1 when the strict hex size is exactly 0. Requirements. Accept only 0 (single digit). Reject 00, 0, 0x0, and non-hex input by returning -1. Input/Output. "0" → 1; "4" → 0; "00" → -1. Hint. Apply the leading-zero rule from concept 2. Concepts. Terminator, strict hex.

Intermediate 1 — Strict size parser

Objective. Implement long parse_chunk_size(const char *hex) that returns the value or -1 on any malformed input. Requirements. Reject leading zeros, signs, whitespace, 0x, trailing garbage, and values over a cap you choose (e.g. 256 KiB). Constraints. No use of atoi. If you use strtoul, check the end pointer. Hint. Confirm the end pointer lands on \r (or string end, per your contract). Concepts. Strict hex parsing, overflow guard.

Intermediate 2 — Reject conflicting length headers

Objective. Given a parsed set of headers, return an error if a message carries both Content-Length and Transfer-Encoding. Requirements. Case-insensitive header name matching; reject duplicates of either header too. Input/Output. Headers with both present → reject; only one present → accept. Hint. This single rule blocks a large class of smuggling. Concepts. Smuggling threat model, defensive parsing.

Challenge — Full streaming decoder with caps

Objective. Extend decode_chunked to consume input from a file descriptor incrementally (read into a fixed buffer) rather than from one in-memory blob, while still enforcing MAX_CHUNK and MAX_TOTAL. Requirements. Handle a chunk whose data spans multiple read() calls; handle a size line split across reads; reject truncated input and any size over the cap; consume trailers and the final CRLF; free all buffers on every path. Constraints. Bounded memory: never hold more than the cap; never trust a length before bounds-checking it. Hint. Keep an explicit parser state (reading-size, reading-data, reading-crlf) so you can pause between reads. Concepts. All of the above plus incremental parsing and resource cleanup.

Summary

Summary

• The format. A chunk is HEXLEN CRLF DATA CRLF. Read the hex size, copy exactly that many data bytes, consume the closing CRLF, repeat. A zero-length chunk (0\r\n\r\n) ends the body; optional trailers may sit before the final blank line.
• The key syntax. Strict hex size + CRLF; memcpy exactly n bytes; realloc to grow the output; a cap-and-overflow guard on every length.
• The common mistakes. Trusting strtol/strtoul without checking the end pointer; scanning data for \r\n instead of using the size; forgetting the final CRLF/trailers; allowing an unbounded total.
• What to remember. The size field is hostile network input. Parse it strictly, bound it, and bounds-check before every copy. Lenient chunked parsing is where HTTP request smuggling lives — so reject conflicting length headers, reject sloppy size lines, and cap both per-chunk and total decoded bytes.

Practice with these exercises

• Parse a chunk size
• Is this the last chunk?
• Decode an HTTP/1.1 chunked body
• Unfold an HTTP/1.1 continuation header