Integer overflow

Overview

Computers store integers in a fixed number of bits — an int is usually 32 bits, a size_t is often 64 bits. A fixed number of bits can only represent a fixed range of numbers. When a calculation produces a result that is too big to fit, the extra bits fall off the top and you are left with a wrong value. This is integer overflow.

The frustrating part is that overflow is silent. There is no exception, no crash, no compiler warning by default. 2000000000 + 2000000000 does not give you four billion — it gives you a negative number, and your program keeps running as if nothing happened.

This lesson builds directly on Data types, where you learned how many bits each type has and the difference between signed and unsigned. That range knowledge is exactly what you need here: overflow is simply arithmetic that escapes the range of its type. In plain language, an integer is like a car odometer with a fixed number of digits — when it maxes out, it rolls over to zero (or, for signed numbers, wraps to a large negative value). The formal term for this whole class of bug is CWE-190: Integer Overflow or Wraparound, and it is one of the most common root causes behind memory-corruption vulnerabilities.

We care about it in a secure coding course because overflow is rarely just a wrong number on screen. It usually feeds a second operation — most dangerously a memory allocation — and turns a harmless-looking miscalculation into a buffer overflow an attacker can exploit.

Why it matters

Integer overflow matters because the wrong number almost never stays contained. It flows downstream into a decision the program trusts completely.

The textbook example is allocation sizing. A program reads a count n from the network or a file, computes n * sizeof(element), and calls malloc with the result. If n is attacker-controlled and large, the multiplication wraps to a small number. malloc happily returns a small buffer. The program then loops n times writing elements — far past the end of that buffer. Now the attacker controls memory beyond the allocation, which is the foundation for code execution, privilege escalation, or a crash.

Real, high-impact vulnerabilities have followed exactly this pattern: the classic OpenSSH "CRC compensation attack detector" bug, numerous image and font parsers (a malformed width/height multiplies to overflow), and countless malloc(count * size) sites that predate calloc's built-in overflow check. Signed overflow adds a second hazard: because it is undefined behaviour, an optimising compiler is allowed to assume it never happens and delete the very if statement you wrote to guard against it. A check that looks correct in the source can vanish from the binary.

Getting this right is a core defensive-programming skill: validate sizes, check arithmetic before you rely on it, and never let untrusted input drive a raw multiplication into an allocator.

Core concepts

1. Fixed width and wraparound

Definition. Every integer type holds values in a fixed range set by its bit width. unsigned char (8 bits) holds 0–255. int (32 bits, typically) holds about -2.1 billion to +2.1 billion. When arithmetic produces a value outside the range, it overflows.

How it works internally. Numbers are stored in binary in a fixed number of bit slots. Addition works bit by bit, carrying into the next slot. When a carry would land in a slot that does not exist (the (N+1)-th bit of an N-bit number), it is simply discarded. What remains is the true result modulo 2^N.

8-bit unsigned addition: 255 + 1

   1 1 1 1 1 1 1 1   (255)
 +             0 0 1   (1)
 ------------------
 [1] 0 0 0 0 0 0 0 0   <- the top carry bit has nowhere to go
  ^
  discarded          result stored = 0000 0000 = 0

When it matters / when not. It matters any time a value can approach the limits of its type — sizes, counts, file offsets, timestamps, accumulators in loops. It does not matter for small, bounded values you fully control (e.g., a loop from 0 to 10 in an int).

Pitfall. Assuming "my numbers are never that big." Attackers pick the inputs. A field you expected to be 3 can arrive as 4,294,967,295.

Knowledge check: An unsigned char currently holds 250. You add 10. What value does it hold afterwards, and why?

2. Signed overflow is undefined behaviour (UB)

Definition. In C, if a signed integer operation produces a result outside the type's range, the behaviour is undefined — the standard imposes no requirement whatsoever.

Plain language. "Undefined" does not mean "wraps to negative." It means the compiler is entitled to assume signed overflow cannot occur and to optimise on that assumption. The program might wrap, might crash, might skip a check, or might behave differently between a debug and a release build.

How it works internally. Optimisers use overflow-freedom as a fact. If you write if (x + 1 < x) to detect signed overflow, the compiler reasons "x + 1 can never be less than x because overflow is impossible," folds the condition to false, and removes your check. Your defence is gone from the machine code even though it is right there in the source.

When to use / avoid. Never rely on signed overflow wrapping. Never write a check that only works after the overflow has already happened on a signed type. Detect it before by comparing against the limits in <limits.h> or by using a builtin.

Pitfall. The post-hoc check int r = a + b; if (r < a) ... is broken for signed types twice over: the addition itself is UB, and the compiler may delete the if.

Knowledge check (find-the-bug): Why can the compiler legally optimise away if (a > 0 && b > 0 && a + b < 0) return ERROR; for signed int a, b?

3. Unsigned overflow is defined: modular wraparound

Definition. For unsigned types, overflow is fully defined: results are reduced modulo 2^N. SIZE_MAX + 1 == 0. (unsigned)0 - 1 == UINT_MAX.

Plain language. Unsigned math is "clock arithmetic." It never invokes UB, but the value is still wrong for your intent. Defined behaviour is not the same as safe behaviour.

How it works internally. The hardware computes the true result and keeps only the low N bits — exactly the diagram above. Because it is defined, you are allowed to check for it after the fact (e.g., if (a + b < a) correctly detects unsigned addition wrap).

When to use / avoid. Prefer size_t for sizes and counts. But remember subtraction: size_t can never be negative, so a - b when b > a produces a gigantic number, not a small negative one — a frequent source of buffer bugs.

Pitfall. for (size_t i = n - 1; i >= 0; i--) is an infinite loop: an unsigned i is always >= 0, and when it "goes below zero" it wraps to SIZE_MAX.

4. The size-calculation attack (CWE-190)

Definition. A multiplication or addition that computes an allocation size overflows, producing a size much smaller than the number of bytes that will actually be written.

How it works internally. Consider a 64-bit size_t and sizeof(T) == 16. If n == 2^60, then n * 16 == 2^64, which wraps to 0. malloc(0) may return a tiny valid pointer. The subsequent loop writes 2^60 elements — catastrophic out-of-bounds write.

requested:  malloc( n * sizeof(T) )  -->  wraps to small buffer
                                          +----------------+
                                          | 0 bytes / tiny |
                                          +----------------+
written:    for i in 0..n:  a[i] = ...        |||||||||||||||||||||||||||||||>
                                          ^ writes run far past the end (heap overflow)

When to use / avoid. Guard every size computation that involves untrusted or large values. calloc(n, size) already does this check internally — prefer it for zero-initialised arrays.

Pitfall. Checking the product against a limit (if (n * size > MAX)) — the product has already overflowed, so the check is meaningless. Always check the inputs: if (n > SIZE_MAX / size).

Knowledge check (predict-the-output): On a system where size_t is 64-bit and sizeof(int) == 4, what does n * sizeof(int) evaluate to when n == (size_t)1 << 62? Is that dangerous for malloc?

Syntax notes

The key patterns are a division-based pre-check and the compiler overflow builtins.

#include <stdint.h>   // SIZE_MAX, uint32_t, ...
#include <limits.h>   // INT_MAX, INT_MIN

/* Pre-check multiply BEFORE doing it: never divide by zero. */
if (size != 0 && n > SIZE_MAX / size) {
    /* would overflow -> reject */
}

/* gcc/clang builtins: compute result, return true on overflow. */
size_t bytes;
if (__builtin_mul_overflow(n, size, &bytes)) {
    /* overflowed; 'bytes' holds the wrapped value, do NOT use it */
}
if (__builtin_add_overflow(a, b, &sum)) { /* overflowed */ }

Notes:

• SIZE_MAX / size is safe only when size != 0; guard the divisor.
• The builtins work for signed and unsigned types and never themselves invoke UB — they are the cleanest option when available.
• calloc(n, size) performs the multiply-with-overflow-check for you and zeroes the memory.

Lesson

Signed vs. unsigned overflow

In C, integer overflow behaves differently depending on the type:

• Signed integer overflow is undefined behaviour (UB). UB means the C standard places no constraints on what happens. The program may crash, produce garbage, or appear to work today and break after a recompile.
• Unsigned integer overflow wraps around modulo 2^N, where N is the number of bits in the type. For example, an 8-bit unsigned value wraps from 255 back to 0.

Why it is dangerous

Either way, the result can be wrong without any warning. A common trap is a size calculation like n * sizeof(T).

If n is large, the multiplication can wrap silently. You then ask for a tiny allocation but later write the full, large number of elements into it. That mismatch is a buffer overflow. This class of bug is tracked as CWE-190 (Integer Overflow or Wraparound).

Defences

Guard the arithmetic before you rely on the result:

• Check before multiplying. Verify n > SIZE_MAX / sizeof(T) and reject the input if it is too large.
• Use compiler builtins. On gcc and clang, __builtin_mul_overflow performs the multiply and reports overflow in one step.
• Use checked-arithmetic wrappers. Helper functions that detect overflow and signal an error.

Code examples

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>

/* Allocate an array of `n` ints, refusing sizes that would overflow.
   Returns NULL on overflow or allocation failure. */
static int *make_int_array(size_t n) {
    /* Guard the size calculation BEFORE multiplying.
       If n > SIZE_MAX / sizeof(int), then n * sizeof(int) would wrap. */
    if (n > SIZE_MAX / sizeof(int)) {
        fprintf(stderr, "refusing: n=%zu would overflow size calculation\n", n);
        return NULL;
    }

    int *a = malloc(n * sizeof(int));   /* now provably safe to compute */
    if (a == NULL && n != 0) {          /* malloc(0) may legitimately return NULL */
        fprintf(stderr, "allocation of %zu ints failed\n", n);
        return NULL;
    }
    for (size_t i = 0; i < n; i++)
        a[i] = (int)i;                  /* safe: buffer really holds n ints */
    return a;
}

/* Same idea using the compiler builtin (gcc/clang). */
static int safe_mul_size(size_t a, size_t b, size_t *out) {
    if (__builtin_mul_overflow(a, b, out))
        return -1;                      /* overflow: *out is wrapped, ignore it */
    return 0;
}

int main(void) {
    /* 1. A normal, in-range request works. */
    size_t good = 5;
    int *arr = make_int_array(good);
    if (arr) {
        printf("allocated %zu ints:", good);
        for (size_t i = 0; i < good; i++) printf(" %d", arr[i]);
        putchar('\n');
        free(arr);                      /* clean up */
    }

    /* 2. A malicious huge request is rejected, not silently wrapped. */
    size_t evil = (SIZE_MAX / sizeof(int)) + 1;  /* just past the safe limit */
    int *bad = make_int_array(evil);
    printf("huge request returned %s (as expected)\n", bad ? "a pointer" : "NULL");
    free(bad);                          /* free(NULL) is safe */

    /* 3. Demonstrate the builtin directly. */
    size_t product;
    if (safe_mul_size(evil, sizeof(int), &product) != 0)
        printf("safe_mul_size detected the overflow\n");
    else
        printf("product = %zu\n", product);

    return 0;
}

What it does. make_int_array refuses any n that would make n * sizeof(int) wrap, then allocates and fills a genuinely n-sized buffer. main allocates a small array (case 1), tries a deliberately overflowing size (case 2), and shows the builtin catching the same overflow (case 3).

Expected output:

allocated 5 ints: 0 1 2 3 4
refusing: n=4611686018427387904 would overflow size calculation
huge request returned NULL (as expected)
safe_mul_size detected the overflow

(The exact n in the "refusing" line depends on your platform's SIZE_MAX and sizeof(int); the value shown assumes a 64-bit size_t and 4-byte int.)

Edge cases. malloc(0) is allowed to return either NULL or a unique freeable pointer, so we only treat NULL as an error when n != 0. free(NULL) is always safe, which is why case 2's cleanup is fine. The builtins require gcc or clang; with other compilers, use the division pre-check instead.

Line by line

1. if (n > SIZE_MAX / sizeof(int)) — the heart of the defence. SIZE_MAX / sizeof(int) is the largest n for which n * sizeof(int) still fits in a size_t. We compare before multiplying, so no overflow ever occurs. Division truncates toward zero, which makes the bound conservative (safe).
2. If the guard triggers, we print a diagnostic and return NULL — the caller learns the request was refused rather than getting a dangerously small buffer.
3. malloc(n * sizeof(int)) runs only after we proved the product fits. The multiply is now genuinely safe.
4. if (a == NULL && n != 0) — distinguishes a real allocation failure from the legal malloc(0) returning NULL.
5. The fill loop writes exactly n ints into a buffer that holds exactly n ints — no out-of-bounds access.
6. In main, case 1 uses good = 5: the guard passes, allocation succeeds, we print and free.
7. Case 2 sets evil = (SIZE_MAX / sizeof(int)) + 1 — the smallest n that would overflow. The guard fires and returns NULL.

8. safe_mul_size(evil, sizeof(int), &product) calls __builtin_mul_overflow, which sets *out to the wrapped value and returns true; we map that to -1, so main reports the overflow was detected.
9. Every allocated pointer is passed to free, and free(NULL) is harmless — so cleanup is uniform and leak-free.

Common mistakes

Mistake 1 — checking the product instead of the inputs.

/* WRONG: the multiply already overflowed before the if runs */
if (n * sizeof(T) > SIZE_MAX) { /* never true for size_t! */ }
T *a = malloc(n * sizeof(T));

n * sizeof(T) is a size_t; it can never exceed SIZE_MAX, so the check is dead code. Fix: check the input against the limit before multiplying: if (n > SIZE_MAX / sizeof(T)) return -1;. Recognise it by asking "could the value in this comparison already have wrapped?"

Mistake 2 — post-hoc check on a signed type.

/* WRONG: a + b is UB on overflow; compiler may delete the if */
int sum = a + b;
if (sum < a) return -1;   /* may be optimised away */

Fix: check before the operation using limits, or use a builtin:

if ((b > 0 && a > INT_MAX - b) || (b < 0 && a < INT_MIN - b)) return -1;
int sum = a + b;   /* now guaranteed in range */
/* or simply: if (__builtin_add_overflow(a, b, &sum)) return -1; */

Recognise it: any check that reads the result of a signed op to decide whether that op overflowed is broken.

Mistake 3 — unsigned subtraction going below zero.

size_t remaining = buf_len - consumed;   /* if consumed > buf_len: HUGE number */
memcpy(dst, src, remaining);             /* massive over-read/over-write */

Fix: validate ordering first: if (consumed > buf_len) return -1;. Recognise it: subtracting unsigned values where the left side could be smaller than the right.

Mistake 4 — mixing signed and unsigned in comparisons.

int len = get_len();                 /* could be negative */
if (len < sizeof(buf)) memcpy(...);  /* len promoted to unsigned: -1 becomes huge */

A negative int compared with a size_t is converted to a giant unsigned value, so the guard passes when it should fail. Fix: validate that len >= 0 as a signed value first, then compare. Enable -Wsign-compare to catch these.

Debugging tips

Compiler help (turn it on).

• -Wall -Wextra surfaces -Wsign-compare and suspicious conversions.
• -fsanitize=undefined (UBSan): instruments the binary so signed overflow is reported at runtime with file and line — invaluable, since signed overflow otherwise leaves no trace.
• -fsanitize=address (ASan): catches the heap buffer overflow that an overflowed size causes, even when the arithmetic itself was unsigned and "defined."
• -ftrapv traps on signed overflow (older; UBSan is generally preferred).

Runtime symptoms to recognise.

• A malloc succeeds but a loop crashes far into it — classic wrapped-size allocation.
• A size that should be small is astronomically large in the debugger — suspect unsigned subtraction wrap.
• A guard that clearly should reject bad input never fires — suspect a signed-overflow check the optimiser removed, or a signed/unsigned comparison.

Steps when it misbehaves.

1. Rebuild with -fsanitize=undefined,address -g -O1 and rerun with the triggering input.
2. Print the intermediate values with %zu (for size_t) right before the allocation — is the size what you expect?
3. Ask: which type is each operand? Is any operand signed and possibly negative? Could the left side of a subtraction be smaller than the right?
4. Replace the raw arithmetic with __builtin_mul_overflow/__builtin_add_overflow and see whether it reports overflow.

Questions to ask: Where does this number come from — is it attacker- or file-controlled? What is its maximum possible value? Is the type wide enough? Did I check before the operation, not after?

Memory safety

Integer overflow is dangerous almost entirely because of what it does to memory operations. The security chain is: untrusted input → overflowed size → undersized allocation → out-of-bounds write → memory corruption → potential code execution. This is CWE-190 (Integer Overflow or Wraparound), frequently feeding CWE-787 (Out-of-bounds Write).

Undefined behaviour concerns specific to this topic.

• Signed overflow is UB. Never depend on it wrapping, and never write a check that only works after it has occurred — the operation is already UB and the optimiser may erase the check. Detect before, with <limits.h> bounds or builtins.
• Unsigned overflow is defined but still wrong. size_t subtraction that goes below zero yields an enormous value; treat it as a validation failure, not a benign wrap.
• Signed↔unsigned conversions. A negative signed length converted to size_t becomes huge and defeats size guards. Validate signedness and range before comparing.

Vulnerability shown, then fixed (lab illustration only).

/* VULNERABLE: attacker controls count; product wraps; heap overflow follows */
record *buf = malloc(count * sizeof(record));
for (size_t i = 0; i < count; i++) buf[i] = read_record();  /* OOB write */

/* FIXED: reject overflowing sizes; prefer calloc's built-in check */
if (count > SIZE_MAX / sizeof(record)) return -1;
record *buf = calloc(count, sizeof(record));   /* checked + zeroed */
if (!buf) return -1;
for (size_t i = 0; i < count; i++) buf[i] = read_record();

Defensive habits. Validate input ranges at the boundary (least privilege for values, too — reject what you cannot safely handle). Use calloc for zero-initialised arrays so the multiply is checked for you. Centralise arithmetic in small safe_add/safe_mul helpers and use them everywhere sizes are computed. Enable UBSan/ASan in CI so overflow bugs surface during testing, not in production.

Real-world uses

Where this bites in real systems. Media and font parsers multiply width * height * bytes_per_pixel from an untrusted header — a malformed image with huge dimensions overflows the size and corrupts the heap; this has produced remotely exploitable bugs in image libraries. Network protocol code multiplies a count field by an element size before allocating. The historical OpenSSH CRC-compensation-detector vulnerability was an integer-overflow-driven allocation bug. Kernels and allocators added checked-multiplication helpers (e.g., kmalloc_array, reallocarray, calloc) precisely because raw malloc(a * b) was such a reliable source of holes.

Professional best practice.

• Beginner: Always use size_t for sizes and counts. Prefer calloc(n, size) over malloc(n * size). Add a n > SIZE_MAX / size guard before any hand-rolled size multiply. Compile with -Wall -Wextra.
• Advanced: Provide and enforce a checked-arithmetic module (safe_mul, safe_add, using __builtin_*_overflow). Run UBSan and ASan in CI and fuzz size-parsing code. Treat every value crossing a trust boundary as hostile: validate its range explicitly, document the maximum, and pick a type wide enough with margin. Use reallocarray where available. Review signed/unsigned conversions at every comparison involving sizes.

Good habits throughout: clear names (element_count, byte_size), a single validation point per input, explicit error returns on rejection, and cleanup (free) on every path — including the rejection paths.

Practice tasks

1. Beginner — Watch it wrap. Write a program that stores 250 in an unsigned char, then adds 10 in a loop of single +1 steps, printing the value each step. Objective: observe the wrap from 255 to 0. Requirements: use %u to print; loop 10 times. Expected: values 251, 252, 253, 254, 255, 0, 1, 2, 3, 4. Hint: promotion means you may need to store back into the unsigned char each step. Concepts: fixed width, unsigned wraparound.

2. Beginner — Safe multiply pre-check. Implement int safe_mul_size(size_t a, size_t b, size_t *out) returning 0 and storing a*b when it fits, or -1 on overflow — using only the division method (b != 0 && a > SIZE_MAX / b). Requirements: handle b == 0 (result 0, no overflow). Test with (3, 4) → 12, and (SIZE_MAX, 2) → -1. Hint: check the divisor before dividing. Concepts: CWE-190 pre-check.

3. Intermediate — Builtin version. Reimplement task 2 using __builtin_mul_overflow, then write a small test that compares its result against the division version for several input pairs and prints PASS/FAIL. Requirements: both functions must agree on every pair. Hint: the builtin returns nonzero on overflow. Concepts: compiler builtins, equivalence testing.

4. Intermediate — Fix the parser. You are given: T *a = malloc(count * sizeof(T)); for (size_t i=0;i<count;i++) a[i]=next(); where count comes from input. Rewrite it to reject overflowing count, prefer calloc, check for allocation failure, and free on all paths. Requirements: no raw unchecked multiply reaches malloc. Constraint: must compile clean under -Wall -Wextra. Hint: calloc already checks the product. Concepts: safe allocation, cleanup.

5. Challenge — Checked signed add. Implement int safe_add_int(int a, int b, int *out) that detects signed overflow without ever performing the overflowing addition, using INT_MAX/INT_MIN from <limits.h>. Requirements: correct for all four sign combinations; return -1 on overflow, else 0 and store the sum. Verify with INT_MAX + 1 (overflow), INT_MIN + -1 (overflow), and 100 + 23 (ok). Constraint: build with -fsanitize=undefined and confirm no UBSan reports fire. Hint: for b > 0, overflow iff a > INT_MAX - b; for b < 0, iff a < INT_MIN - b. Concepts: signed overflow is UB, check-before-operate.

Summary

Main concepts. Integers are fixed-width, so arithmetic that escapes the type's range overflows. Signed overflow is undefined behaviour — the compiler may assume it never happens and delete your checks. Unsigned overflow is defined (modulo 2^N) but still produces a wrong value, and unsigned subtraction below zero yields a huge number. The headline danger is CWE-190: an overflowed size calculation (n * sizeof(T)) leads to a tiny allocation followed by a large write — a heap buffer overflow attackers exploit.

Most important syntax. Check inputs before multiplying: if (size != 0 && n > SIZE_MAX / size) reject;. Or use the builtins: __builtin_mul_overflow(a, b, &out) / __builtin_add_overflow(a, b, &out). Prefer calloc(n, size) for arrays — its multiply is checked and the memory is zeroed.

Common mistakes. Checking the product instead of the inputs; post-hoc checks on signed types (UB + optimised away); unsigned subtraction underflow; signed/unsigned comparison converting a negative length to a huge unsigned value.

What to remember. Overflow is silent, and the wrong number almost always flows into an allocation or a copy. Treat every size that touches untrusted input as hostile: validate its range, check arithmetic before you rely on it, prefer calloc/builtins, and turn on -Wall -Wextra -fsanitize=undefined,address while testing.

Practice with these exercises

• Overflow-checked multiply
• Overflow-checked size addition