Subtract two ints with overflow detection

Challenge

Subtract two ints, detecting overflow before it happens. Signed subtraction overflows in different cases than addition, and audit code rarely checks it.

Task

Implement int sub_safe(int a, int b, int *out) that stores a - b in *out only when it fits in an int.

Check before subtracting, using:

• overflow above INT_MAX: b < 0 && a > INT_MAX + b
• overflow below INT_MIN: b > 0 && a < INT_MIN + b

Input

• a, b: the minuend and subtrahend.
• out: where the difference is written on success.

Output

Returns 0 on success (with *out = a - b), or -1 if the subtraction would overflow int.

Example

int out;
sub_safe(5, 3, &out)         ->   0, out == 2
sub_safe(INT_MAX, -1, &out)  ->   -1   (overflows up)
sub_safe(INT_MIN, 1, &out)   ->   -1   (overflows down)
sub_safe(0, 0, &out)         ->   0, out == 0
sub_safe(-5, -3, &out)       ->   0, out == -2

Edge cases

• INT_MAX - (-1) overflows up; INT_MIN - 1 overflows down.
• 0 - 0 succeeds.

Rules

• Check before subtracting; no GCC/Clang built-ins. Portable C only.

Why this matters

Signed subtraction overflows in different cases than addition. Audit code rarely checks; the bug class is real in size calculations.

Input format

Two ints a, b, and an output pointer out.

Output format

0 on success (with *out = a - b); -1 on overflow (with *out unchanged).

Constraints

Check before subtracting; no __builtin_* helpers; portable C.

Starter code

#include <limits.h>
int sub_safe(int a, int b, int *out) { /* TODO */ return -1; }

Common mistakes

Doing a - b first and then comparing — undefined.

Edge cases to handle

INT_MIN - 1; INT_MAX - (-1); 0 - 0; INT_MIN - INT_MIN.

Complexity

O(1).

Background lessons

• Integer overflow
• Memory safety — the whole bug class

Up next

• Add two ints with overflow detection
• Multiply two ints with overflow detection