Linux System Programming · intermediate · ~12 min
- By the end you can define a race condition precisely: unsynchronised concurrent access to shared memory where at least one access is a write. - By the end you can explain why an operation like `counter++` is not atomic, and trace how two threads interleave its steps to lose an update. - By the end you can predict the conditions under which a race triggers (shared mutable state + write + no ordering) and reason about why it is intermittent. - By the end you can recognise that a data race is undefined behaviour in C, not merely a wrong answer, and why `volatile` does not fix it. - By the end you can detect races with ThreadSanitizer and describe the categories of fix (atomics, mutexes) that later lessons build on.
You already know how to launch threads with pthread_create and collect their results with pthread_join (from Thread return values). Joining gives you one kind of coordination: it tells the main thread when a worker has finished. This lesson is about what happens before that join, while several threads run at the same time and touch the same variable. That is where race conditions live.
A race condition is the single most important hazard in concurrent C. It is subtle because the buggy program usually looks correct, compiles without warnings, and even produces the right answer most of the time. This lesson builds directly on your thread knowledge: same pthread_create/pthread_join skeleton you already use, but now with shared mutable state, so you can see exactly where and why concurrency goes wrong. The fix (mutexes) is the next lesson; here the goal is to recognise and predict the problem cold.
Race conditions cause the worst class of production bugs: nondeterministic, load-dependent, and nearly impossible to reproduce on demand. A financial service that double-spends under load, a counter that drifts, a linked list that corrupts one time in ten thousand — these are races. On the security side, Time-Of-Check-To-Time-Of-Use (TOCTOU) races let an attacker slip a malicious file or permission change into the gap between a check and the action that trusts it, turning a safe-looking access()-then-open() into a privilege escalation. Because a data race is undefined behaviour, the compiler may optimise your code in ways that make the bug even stranger — so getting this right is a correctness and a safety requirement.
A data race occurs when all three of these are true at once:
Remove any one of the three and the race disappears. Two threads only reading a shared value is fine. Two threads writing to different variables is fine. Two threads writing the same variable but separated by a mutex is fine. It is the combination — shared + write + unordered — that is the bug.
The outcome of a data race depends on which thread's memory operations happen to land first, which depends on the OS scheduler, CPU load, cache state, and luck. That is why races are intermittent: the same binary gives different answers on different runs.
counter++ is three steps, not oneThe canonical race is a shared counter. The source looks atomic, but the CPU sees a read-modify-write sequence:
counter++; becomes: 1. load R <- counter (read)
2. add R <- R + 1 (modify)
3. store counter <- R (write)
With two threads there is no guarantee those three steps run back-to-back. Here is a losing interleave, with counter starting at 41:
time Thread A Thread B counter in memory
| load R=41 41
| load R=41 41
| add R=42 41
| add R=42 41
| store counter=42 42
v store counter=42 42 <- should be 43!
Both threads read 41, both compute 42, both write 42. Two increments happened in the source; only one stuck. This is a lost update. Run four threads incrementing 200,000 times each and the final value is dramatically below the 800,000 you expected.
Knowledge check: two threads each run x = x + 1 once on a shared x that starts at 0. What are the possible final values of x?
1or2. If the two increments happen to run without overlapping, you get2. If they interleave (both read0, both write1), one update is lost and you get1. You can never get3, and — barring other UB — never0. The fact that both answers are possible from the same code is the defining symptom of a race.
It is tempting to think a race just means "the counter is a bit off." In C11 it is worse: a data race is undefined behaviour. The standard places no constraints on a program that contains one. The compiler is allowed to assume races never happen and optimise accordingly — for example, hoisting a shared read out of a loop into a register, so a change made by another thread is never observed at all. This is why you cannot reason about a racy program by "reading the assembly": the assembly you imagine may not be the assembly you get.
volatile is not the fixTwo distinct problems hide inside a race:
counter++ case above).long on a 32-bit target, or an unaligned field) can be written in two halves; a reader can catch it half-updated.| Tool | Guarantees atomicity? | Guarantees visibility/ordering? | Correct fix for a race? |
|---|---|---|---|
| plain variable | no | no | no |
volatile |
no | no (only stops the compiler caching in a register) | no |
_Atomic / atomic_fetch_add |
yes | yes | yes (for single-variable ops) |
pthread_mutex_t |
yes (for the whole critical section) | yes | yes (next lesson) |
volatile is the most common false fix. It tells the compiler "re-read this from memory each time" — useful for memory-mapped hardware registers and sig_atomic_t signal flags — but it provides no atomicity and no inter-thread ordering. A volatile int counter; counter++; is still a three-step race. The real fixes are C11 atomics (_Atomic, <stdatomic.h>) for single-variable operations, or a mutex to protect a whole critical section.
Races are not only about counters. A Time-Of-Check-To-Time-Of-Use race happens when a program checks a condition and then acts on it, and something changes in the gap:
access("/tmp/data", W_OK) <- check: "am I allowed to write this?"
... attacker swaps /tmp/data for a symlink to /etc/passwd ...
open("/tmp/data", O_WRONLY) <- use: now writing the wrong file
The defensive fix is to eliminate the gap: operate on a stable handle (open once, then fstat/fchmod the file descriptor rather than re-resolving the path), use atomic flags like O_CREAT | O_EXCL, and avoid check-then-act on shared filesystem paths. Same principle as the counter: don't split an operation that must be indivisible.
atomic_long safe = 0; — declares an atomic integer (from <stdatomic.h>, C11). Operations on it are indivisible and establish inter-thread ordering. Equivalent spelling: _Atomic long.
long atomic_fetch_add(volatile atomic_long *obj, long arg); — atomically adds arg to *obj and returns the previous value, as one indivisible step. No lost updates even under contention. There are matching atomic_fetch_sub, atomic_load, atomic_store, and atomic_compare_exchange_*.
long atomic_load(const volatile atomic_long *obj); — atomically reads the current value. Use this to read an atomic; do not read it with a plain access.
int pthread_create(pthread_t *thread, const pthread_attr_t *attr, void *(*start)(void *), void *arg); — starts start(arg) on a new thread; returns 0 on success or an errno-style code (it does not set errno). You already know this from the prerequisite.
int pthread_join(pthread_t thread, void **retval); — blocks until thread finishes; returns 0 or an error code. Joining a thread creates a happens-before edge, so all that thread's writes are visible afterwards — this is why reading racy in main after the joins is itself safe.
Build flag: -fsanitize=thread — compile and link with this to enable ThreadSanitizer, which reports the exact two accesses that race.
Link flag: -lpthread — required to pull in the pthreads implementation.
A race condition happens when two or more threads access the same memory, at least one of them writes to it, and they do not coordinate that access.
The outcome then depends on timing: which thread happens to run first. In C, this is undefined behaviour ("UB" — the C standard places no constraints on what the program may do). The compiler is even allowed to optimise the code as if the race could never occur.
The simplest demonstration is two threads incrementing the same counter.
int counter = 0;
/* thread A and thread B both run: */
counter++; /* read counter, add 1, write counter — three steps */
The key insight is that counter++ is not a single action. It is three steps:
counter.Because the threads are not coordinated, these steps can interleave:
0.1.1.One increment is lost. Run two threads that each increment 1,000,000 times, and the final value will be noticeably smaller than 2,000,000.
Sanitisers catch this quickly. Build with ThreadSanitizer (TSan):
gcc -fsanitize=thread
Always test concurrent code with TSan.
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <stdatomic.h>
#define THREADS 4
#define PER_THREAD 200000
/* Two counters driven by identical work: one with a plain
* read-modify-write (racy), one with an atomic add (correct). */
static long racy = 0; /* UNSYNCHRONISED shared write -> data race */
static atomic_long safe = 0; /* indivisible updates, no race */
static void *worker(void *arg) {
(void)arg;
for (int i = 0; i < PER_THREAD; i++) {
racy++; /* THREE steps: load, add, store */
atomic_fetch_add(&safe, 1); /* ONE indivisible step */
}
return NULL;
}
int main(void) {
pthread_t t[THREADS];
for (int i = 0; i < THREADS; i++) {
if (pthread_create(&t[i], NULL, worker, NULL) != 0) {
perror("pthread_create");
return EXIT_FAILURE;
}
}
for (int i = 0; i < THREADS; i++)
pthread_join(t[i], NULL); /* now all worker writes are visible */
long expected = (long)THREADS * PER_THREAD;
long got_safe = (long)atomic_load(&safe);
printf("expected = %ld\n", expected);
printf("racy counter = %ld (lost %ld updates)\n", racy, expected - racy);
printf("atomic counter = %ld (lost %ld updates)\n", got_safe, expected - got_safe);
if (racy != expected)
puts("\n-> The racy counter lost updates: that is the race condition.");
else
puts("\n-> Racy counter matched THIS run; rerun and it will drift.");
return 0;
}
#include <stdatomic.h> — pulls in the C11 atomics used for the safe counter; <pthread.h> gives the thread API you already know.static long racy = 0; — an ordinary shared variable. Every thread writes it with no coordination, so it satisfies all three race conditions: shared, written, unordered.static atomic_long safe = 0; — the control case. Same sharing, but the type makes each update indivisible, so it is not a race.racy++; — the bug. The compiler emits load/add/store; two threads' sequences interleave and lose updates, exactly as in the timeline diagram.atomic_fetch_add(&safe, 1); — the same +1, but performed as a single hardware-level atomic operation. No interleave can split it.pthread_create loop — spawns THREADS workers all running the same function on the same globals, maximising contention so the race shows up reliably. The != 0 check follows the pthreads error convention (return code, not errno).pthread_join loop — waits for every worker. Beyond "wait", the join establishes a happens-before relationship, which is what makes it safe for main to read racy and safe afterwards.expected = THREADS * PER_THREAD — the arithmetic ceiling both counters should reach (800,000 with the defaults).printfs — print expected vs. actual side by side. racy comes out well below expected (lost updates); safe hits it exactly, every run.if — narrates the result. On a busy multicore machine racy essentially never matches; the else branch exists only for the rare quiet single-core case, and even then rerunning makes it drift.volatile long counter = 0;
/* ... in each thread ... */
counter++;
Why it breaks: volatile only stops the compiler from caching the value in a register between accesses. It gives no atomicity and no inter-thread ordering, so counter++ is still a load-add-store that interleaves and loses updates. The race is fully intact.
atomic_long counter = 0;
/* ... in each thread ... */
atomic_fetch_add(&counter, 1);
/* "x86 increments are atomic, so this is fine" */
long counter = 0;
counter++;
Why it breaks: Even on x86, a plain counter++ compiles to separate load and store instructions (only lock-prefixed forms are atomic), and the C standard calls the unsynchronised access UB regardless of CPU — so the compiler may optimise it in surprising ways. 'Works on my machine' is not a guarantee.
atomic_long counter = 0;
atomic_fetch_add(&counter, 1); /* emits the lock-prefixed / LL-SC form */
atomic_long safe = 0;
long snapshot = safe; /* plain read of an atomic */
Why it breaks: Mixing a plain access with atomic accesses to the same object reintroduces a data race and defeats the point. Some compilers accept the implicit conversion, hiding the mistake.
atomic_long safe = 0;
long snapshot = atomic_load(&safe);
if (access(path, W_OK) == 0) { /* check */
int fd = open(path, O_WRONLY); /* use, later */
}
Why it breaks: TOCTOU race: between the check and the open, the path can be swapped for a symlink to a sensitive file, so the write lands somewhere the process was never authorised to touch.
int fd = open(path, O_WRONLY | O_NOFOLLOW);
if (fd >= 0) {
/* check permissions on the fd itself with fstat, then use fd */
}
cc -std=c11 -fsanitize=thread -g prog.c -lpthread. TSan instruments memory accesses and prints the two racing accesses with both stack traces and the variable involved. It finds races even on runs where the output happens to look correct.for i in $(seq 200); do ./prog; done) and watch the racy counter drift between runs — nondeterministic output across identical runs is the signature.valgrind --tool=helgrind ./prog is an alternative race/lock-order detector when you cannot rebuild with TSan; it is slower but needs no recompilation.printf debugging lies here. Adding prints changes timing and often hides the race (a Heisenbug). Prefer sanitizers over print statements for concurrency bugs.strace/ltrace help with TOCTOU-style races by showing the exact syscall sequence (check then use) an attacker could exploit.long on some 32-bit ABIs, or a misaligned field) can be written in two parts; a concurrent reader may observe a half-written, meaningless value. Atomics prevent tearing; plain wide variables do not.volatile does not solve this for inter-thread work; only atomics and mutexes establish the required happens-before ordering.main after pthread_join is well-defined because the join creates a happens-before edge — all the joined thread's writes are complete and visible. The race exists only during concurrent execution, not after.atomic_fetch_add/atomic_fetch_sub.pthread_once, or an atomic flag) — a textbook source of double-initialisation bugs.O_CREAT|O_EXCL for exclusive create, and *at() syscalls with directory fds. CWE-362 (race condition) and CWE-367 (TOCTOU) track these.Reproduce it. Take the shared-counter program, remove the atomic counter, run it 50 times in a loop, and record the final racy value each time. Confirm it varies and is almost always below the expected total.
Predict then measure. For two threads each incrementing a shared int from 0 exactly once, write down every possible final value before running. Then instrument the code to force both orderings (e.g. with small sleeps) and confirm your prediction of 1 vs 2.
Prove volatile fails. Change the racy counter's type to volatile long, rerun the loop, and show the counter still loses updates. Write one sentence explaining why volatile did not help.
Catch it with a tool. Rebuild the racy program with -fsanitize=thread and capture TSan's report. Identify from the output the two lines that race and the variable named.
Design a TOCTOU repro (lab-only, localhost/temp files). Write a program that does access() then open() on a temp path with a deliberate delay between them, and a second process that swaps the path during the gap, to observe the wrong file being used. Then rewrite it using open-once-and-fstat-the-fd (with O_NOFOLLOW) and show the swap no longer changes the target.
counter++ is a read-modify-write (load, add, store); interleaving two of them loses updates. The result is nondeterministic across runs.volatile is not a fix: it stops register caching but gives no atomicity and no inter-thread ordering. Use _Atomic/atomic_fetch_add for single variables, and a mutex (next lesson) for multi-step critical sections.-fsanitize=thread) and by looping the program to watch output drift; printf tends to hide races.pthread_join creates a happens-before edge, so reading shared state after the join is safe.