Safe Penetration Testing Labs · intermediate · ~15 min
By the end of this lesson you will be able to: - Read the **TCP flags byte** out of a packet header and identify the SYN, NULL, FIN, and XMAS scan signatures using bitwise masks. - Explain why a normal TCP exchange (which always has **ACK** set) never matches a scan signature, and use that fact to avoid false positives. - Compute **port fan-out** — the number of *distinct* destination ports a single source IP contacts — and recognise it as the behavioural signature of a scan. - Combine a packet-level signal and a behavioural signal into a single, more reliable detection heuristic. - Write safe C that parses fixed, in-memory fixtures with correct **bounds checks** before touching any byte. - Apply defensive habits: validate buffer lengths, log detections without leaking sensitive data, and keep all work inside an authorized lab.
A port scan is the reconnaissance step an attacker (or a defender's own auditing tool) uses to find out which network services a machine is running. The scanner sends many small probes to many ports and watches which ones answer. Each open port is a potential door; the scan is someone rattling all the doorknobs to see which ones are unlocked.
This lesson teaches you to detect that activity — the defensive side. It is the C foundation under what a tiny intrusion-detection system (IDS) does. You will not build a scanner; you will build the logic that recognises one from evidence it leaves behind.
Scans leave two independent kinds of evidence, and a good detector reads both:
This lesson builds directly on two prerequisites. From bitwise operators you already know how to isolate specific bits of a byte with & and a mask — that is exactly how we read individual TCP flags. From C strings you know how to walk a char buffer and pull out tokens — that is how we parse a text connection log into source IPs and ports. Here we put both skills to work on a real security task.
With the vocabulary in place: a flag signature is a specific value of the TCP flags byte; fan-out is the count of distinct destination ports per source; a threshold is the line above which we call the behaviour a scan; and false positive / false negative are the two ways any detector can be wrong.
Every server exposed to a network is scanned constantly — automated bots sweep the internet looking for open, vulnerable services within minutes of a host coming online. Detecting that early reconnaissance is one of the cheapest, highest-value things a defender can do, because a scan almost always precedes an actual attack. If you can see the scan, you can block the source, raise an alert, or tighten a firewall rule before anything is exploited.
Real detectors combine packet-level and behavioural signals on purpose, and the reason is precision:
The same skills underpin firewalls (iptables/nftables rate-limit rules), IDS/IPS engines like Snort and Suricata, and cloud security monitoring. Learning the logic in C — close to the bytes — makes the higher-level tools far less mysterious.
[ Authorization & ethics ]
Port scanning and scan detection are taught here for DEFENSE only.
- Run probes ONLY against systems you own or are explicitly authorized to test
(localhost, a container, a deliberately vulnerable lab VM, or a CTF range).
- Scanning third-party systems without written permission is illegal in many
jurisdictions, even if you cause no damage.
- Everything in this lesson operates on STATIC, pre-built fixtures. No packets
are sent. Keep it that way until you have authorization to do otherwise.
Definition. A TCP header carries a set of single-bit control flags. The six classic flags — URG, ACK, PSH, RST, SYN, FIN — live in the low 6 bits of byte 13 (0-indexed) of the TCP header.
Plain-language explanation. Each flag is a yes/no switch that says something about the packet: "this starts a connection" (SYN), "this acknowledges data" (ACK), "this ends the connection" (FIN), and so on. A single byte holds all of them, one per bit.
How it works internally. The byte packs the flags like this (high bits first):
TCP flags byte (offset 13 in the TCP header)
bit: 7 6 5 4 3 2 1 0
+----+----+----+----+----+----+----+----+
| -- | -- |URG |ACK |PSH |RST |SYN |FIN |
+----+----+----+----+----+----+----+----+
hex: 0x20 0x10 0x08 0x04 0x02 0x01
(The top two bits are reserved/extended flags; ignore them for scan ID.)
To test whether SYN is set, you mask with its bit value: byte & 0x02. This is exactly the bitwise-AND idea from the bitwise operators lesson.
Structure — the scan signatures.
| Scan | Flags set | Flags byte |
|---|---|---|
| SYN | SYN | 0x02 |
| NULL | (none) | 0x00 |
| FIN | FIN | 0x01 |
| XMAS | FIN + PSH + URG | 0x29 |
NULL, FIN, and XMAS are "stealth" scans: they send weird flag combinations hoping a firewall or host will reveal port state in how it (doesn't) respond.
When to use / not use this signal. Flag signatures are great for catching stealth scans cheaply — one mask per packet. They are not enough on their own, because the most common scan, the SYN scan, uses a flags byte (0x02) that overlaps with the very first packet of a normal connection. So flag-matching catches the exotic scans but needs a partner signal for the common one.
Common pitfall. Forgetting to mask off the reserved/high bits, or comparing the whole byte with == when extra bits may be set. Use a mask to look only at the bits you care about.
Knowledge check (predict the output): A packet has flags byte
0x12. Which flags are set, and is this a scan signature or normal traffic? (Hint:0x12 = 0x10 | 0x02.)
Definition. After the opening SYN, every packet in a legitimate TCP conversation has the ACK bit (0x10) set.
Explanation. TCP is reliable, so once a connection is established both sides acknowledge what they receive. That means real data and teardown packets carry ACK. The SYN scan's lone-SYN and the stealth scans' no-ACK combinations are precisely what does not appear in healthy traffic.
Why it helps detection. It gives a clean rule: a packet whose flags match 0x00, 0x01, or 0x29 is never normal — flag it. A bare 0x02 (SYN) is ambiguous (it could be a genuine new connection), which is exactly why we lean on behaviour for that case.
Pitfall. Treating a SYN-ACK (0x12) reply as suspicious. It has ACK set and is the normal answer to a connection request — do not flag it.
Knowledge check (explain in your own words): Why does the existence of the ACK flag make NULL, FIN, and XMAS scans easy to detect but a SYN scan hard to detect from flags alone?
Definition. Fan-out is the number of distinct destination ports a single source IP contacts within a time window.
Explanation. A real client talks to a small set of ports (web to 443, mail to 25, etc.). A scanner deliberately touches many ports to map what is open. So a high distinct-port count from one source is the behaviour of scanning, no matter how each individual packet is shaped.
How it works internally. For each source IP you keep a set of the ports it has touched and count the set's size. Repeated hits to the same port add nothing — only new ports increase the count.
Fan-out per source IP
src 10.0.0.5 -> ports { 22, 80, 443, 8080, 3306, 25, 110, ... } (count = 11) SCAN
src 10.0.0.9 -> ports { 443, 443, 443 } (count = 1) normal
^ duplicates do not raise the count
Structure — the detection. Compare each source's distinct-port count to a threshold (e.g. "more than 10 ports"). At or above it → scanner. The threshold trades sensitivity against false positives.
When to use / not use. Fan-out catches scans regardless of flag tricks, including plain SYN scans. It can mislabel legitimately busy sources, so pick the threshold for your environment and ideally require a flag signal too.
Pitfall. Counting total connections instead of distinct ports. A client hammering port 443 a thousand times is not a scanner; counting raw hits would wrongly flag it.
Knowledge check (find-the-bug): A learner detects scanners by counting how many log lines each source IP produced and flagging any source with more than 10 lines. Why will this both miss real scans and flag innocent clients?
Two building blocks. First, isolating a TCP flag with a mask (bitwise-AND, then test against zero):
#include <stdint.h>
#define F_FIN 0x01
#define F_SYN 0x02
#define F_PSH 0x08
#define F_ACK 0x10
#define F_URG 0x20
uint8_t flags = tcphdr[13]; /* byte 13 holds the 6 control flags */
int syn_only = (flags == F_SYN); /* exact match: only SYN set */
int has_ack = (flags & F_ACK) != 0; /* test a single bit with a mask */
Second, the safe access pattern: check the length before you index. The TCP header's flags live at offset 13, so the buffer must be at least 14 bytes:
if (n < 14) return -1; /* too short to contain a flags byte -> reject */
uint8_t flags = tcphdr[13] & 0x3F; /* keep only the low 6 flag bits */
For the behavioural side, you walk a text log line by line and split each line into a source IP and a port. With fixed fixtures you can use the standard string tools from the C strings lesson (strtok_r, strchr, strtol) — always on a buffer whose length you know.
Port scans leave two kinds of evidence:
A good detector reads both.
Every TCP packet carries a flags byte. Scans show up in its low 6 bits.
| Scan | Flags set | Hex |
|---|---|---|
| SYN | SYN | 0x02 |
| NULL | (none) | 0x00 |
| FIN | FIN | 0x01 |
| XMAS | FIN+PSH+URG | 0x29 |
Normal traffic always has the ACK flag set. For example, a SYN-ACK reply is 0x12. Because ACK is always present in legitimate exchanges, real traffic never matches a scan signature.
Fan-out means the number of distinct destination ports a single source IP contacts.
tcp_scan_type(...) — classify the flag byte of a single header.count_scanners(...) — from a static connection log, count the source IPs whose distinct-port fan-out exceeds a threshold.This is not a live scanner or packet sniffer. Every input is a static, pre-built fixture.
Below is a complete, compilable C11 program. It classifies a TCP flags byte and counts distinct-port fan-out per source from a small in-memory log, then reports which sources crossed a threshold. Everything is a static fixture — no network access.
#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <stdlib.h>
/* ---- TCP flag bit values (low 6 bits of header byte 13) ---- */
#define F_FIN 0x01
#define F_SYN 0x02
#define F_RST 0x04
#define F_PSH 0x08
#define F_ACK 0x10
#define F_URG 0x20
/* Scan type codes returned by classify_scan(). */
enum { SCAN_NONE = 0, SCAN_SYN, SCAN_NULL, SCAN_FIN, SCAN_XMAS };
/* Classify the flags byte of one TCP header.
Returns -1 if the buffer is too short to hold a flags byte. */
static int classify_scan(const uint8_t *tcphdr, size_t n) {
if (tcphdr == NULL || n < 14) return -1; /* bounds check first */
uint8_t f = tcphdr[13] & 0x3F; /* keep only the 6 flag bits */
if (f & F_ACK) return SCAN_NONE; /* ACK set => part of a real exchange */
if (f == 0x00) return SCAN_NULL;
if (f == F_FIN) return SCAN_FIN;
if (f == (F_FIN|F_PSH|F_URG)) return SCAN_XMAS; /* 0x29 */
if (f == F_SYN) return SCAN_SYN; /* ambiguous: confirm via fan-out */
return SCAN_NONE;
}
/* Per-source fan-out tracking. Small fixed table is fine for a fixture. */
#define MAX_SRC 16
#define MAX_PORTS 64
struct src_entry {
char ip[40]; /* fits an IPv4 or IPv6 text address */
int ports[MAX_PORTS]; /* distinct ports seen for this source */
int nports;
};
static struct src_entry *find_or_add(struct src_entry *t, int *n, const char *ip) {
for (int i = 0; i < *n; i++)
if (strcmp(t[i].ip, ip) == 0) return &t[i];
if (*n >= MAX_SRC) return NULL; /* table full -> caller handles */
struct src_entry *e = &t[*n];
snprintf(e->ip, sizeof e->ip, "%s", ip); /* bounded copy, always NUL-terminated */
e->nports = 0;
(*n)++;
return e;
}
/* Record a port for a source only if it is new (distinct). */
static void add_distinct_port(struct src_entry *e, int port) {
for (int i = 0; i < e->nports; i++)
if (e->ports[i] == port) return; /* duplicate -> ignore */
if (e->nports < MAX_PORTS) e->ports[e->nports++] = port;
}
int main(void) {
/* --- Part 1: classify some flag bytes --- */
const char *names[] = {"NONE","SYN","NULL","FIN","XMAS"};
uint8_t hdr[14] = {0}; /* zeroed 14-byte TCP header */
uint8_t samples[] = {0x02, 0x00, 0x01, 0x29, 0x12};
for (size_t i = 0; i < sizeof samples; i++) {
hdr[13] = samples[i];
int t = classify_scan(hdr, sizeof hdr);
printf("flags 0x%02X -> %s\n", samples[i], t < 0 ? "ERR" : names[t]);
}
/* --- Part 2: fan-out from a static connection log --- */
/* Format per line: "<src-ip> <dst-port>" */
char log[] =
"10.0.0.5 22\n10.0.0.5 80\n10.0.0.5 443\n10.0.0.5 3306\n"
"10.0.0.5 25\n10.0.0.5 110\n10.0.0.5 143\n10.0.0.5 8080\n"
"10.0.0.5 53\n10.0.0.5 5900\n10.0.0.5 23\n"
"10.0.0.9 443\n10.0.0.9 443\n10.0.0.9 443\n";
const int threshold = 10;
struct src_entry table[MAX_SRC];
int nsrc = 0;
char *save = NULL;
for (char *line = strtok_r(log, "\n", &save);
line != NULL;
line = strtok_r(NULL, "\n", &save)) {
char *sp = strchr(line, ' ');
if (sp == NULL) continue; /* malformed line -> skip */
*sp = '\0'; /* split ip | port */
const char *ip = line;
int port = (int)strtol(sp + 1, NULL, 10);
if (port <= 0 || port > 65535) continue; /* validate port range */
struct src_entry *e = find_or_add(table, &nsrc, ip);
if (e == NULL) { fprintf(stderr, "source table full\n"); continue; }
add_distinct_port(e, port);
}
int scanners = 0;
for (int i = 0; i < nsrc; i++) {
int fanout = table[i].nports;
int is_scan = fanout > threshold;
printf("src %-12s fan-out=%-2d %s\n",
table[i].ip, fanout, is_scan ? "SCAN" : "ok");
if (is_scan) scanners++;
}
printf("scanners detected: %d\n", scanners);
return 0;
}
What it does. Part 1 runs five flag bytes through classify_scan. Part 2 parses a static log, tallies distinct destination ports per source, and reports any source whose fan-out exceeds 10.
Expected output:
flags 0x02 -> SYN
flags 0x00 -> NULL
flags 0x01 -> FIN
flags 0x29 -> XMAS
flags 0x12 -> NONE
src 10.0.0.5 fan-out=11 SCAN
src 10.0.0.9 fan-out=1 ok
scanners detected: 1
Edge cases to note: a header shorter than 14 bytes returns -1 (handled before any indexing); duplicate ports for 10.0.0.9 collapse to a fan-out of 1; ports outside 1..65535 and lines without a space are skipped; the source table and per-source port array are bounded, and overflow is handled rather than ignored.
Part 1 — flag classification. classify_scan first rejects a NULL pointer or a buffer shorter than 14 bytes, so it never reads byte 13 out of bounds. It then masks with & 0x3F to keep only the six flag bits. The very first test is f & F_ACK: if ACK is set the packet belongs to a real exchange, so we return SCAN_NONE immediately — this is what makes 0x12 (SYN+ACK) come back as NONE. After ACK is ruled out, exact-match tests pick out NULL (0x00), FIN (0x01), XMAS (0x29), and finally lone SYN (0x02).
Trace of the five samples:
| Input | f & 0x3F |
ACK set? | Result |
|---|---|---|---|
| 0x02 | 0x02 | no | SYN |
| 0x00 | 0x00 | no | NULL |
| 0x01 | 0x01 | no | FIN |
| 0x29 | 0x29 | no | XMAS |
| 0x12 | 0x12 | yes | NONE |
Part 2 — fan-out. strtok_r walks the log one line at a time; the _r (re-entrant) form keeps its state in save instead of a hidden global, which is the safe choice. For each line, strchr(line, ' ') finds the space; writing '\0' there splits the line into an IP string and a port string in place. strtol parses the port, and we reject anything outside 1..65535 before using it.
find_or_add does a linear search for the source IP; if it is new and the table has room, snprintf copies the IP with a hard size limit so it is always NUL-terminated. add_distinct_port scans the source's existing ports and only appends a port it has not seen — this is what makes 10.0.0.9's three hits to 443 count as fan-out 1.
State of the two sources after the loop:
| Source | distinct ports | fan-out |
|---|---|---|
| 10.0.0.5 | 22,80,443,3306,25,110,143,8080,53,5900,23 | 11 |
| 10.0.0.9 | 443 | 1 |
Finally each source's fan-out is compared with threshold = 10. Only 10.0.0.5 exceeds it, so scanners ends at 1.
Mistake 1 — comparing the whole flags byte with == without masking.
/* WRONG: extra/reserved bits make the comparison miss real scans */
if (tcphdr[13] == 0x02) return SCAN_SYN;
If any reserved or extended bit is set, the equality fails even though SYN is set. Mask first, then compare: uint8_t f = tcphdr[13] & 0x3F; and test f.
Mistake 2 — indexing before checking the length.
/* WRONG: reads byte 13 even if the caller passed a 4-byte buffer */
uint8_t f = tcphdr[13];
This is an out-of-bounds read (undefined behaviour) whenever n < 14. Always guard: if (tcphdr == NULL || n < 14) return -1; before the access. Recognise it when a fuzz/short-input test crashes or AddressSanitizer reports a heap-buffer-overflow.
Mistake 3 — counting total hits instead of distinct ports.
/* WRONG: a busy client to one port looks like a scanner */
if (++hits_for_source > threshold) flag_as_scanner();
A monitoring agent that pings port 443 a thousand times would be flagged (false positive), while a slow, careful scanner that touches each port once might slip under a hit-based threshold tuned for volume. Count the size of the set of distinct ports instead.
Mistake 4 — flagging SYN-ACK replies. Treating 0x12 as suspicious mislabels the normal answer to a connection request. Because ACK is set, it must be classified as normal — which is exactly why classify_scan tests ACK first.
Mistake 5 — trusting strtol blindly. Forgetting the range check lets a corrupt log line inject a nonsensical port (negative, zero, or > 65535) into your table. Validate port > 0 && port <= 65535 before recording it.
Compiler errors.
implicit declaration of function 'strtok_r' / 'snprintf' — make sure you compiled with C11 (-std=c11) and included <string.h> and <stdio.h>. On strict POSIX setups you may need #define _POSIX_C_SOURCE 200809L above the includes for strtok_r.comparison is always true warnings often mean you compared an unsigned value the wrong way or used = instead of == in a condition. Build with -Wall -Wextra and read every warning.Runtime errors.
AddressSanitizer: heap-buffer-overflow reading the header → you indexed before checking n >= 14, or your fixture buffer is smaller than you thought. Compile with -fsanitize=address,undefined -g and rerun; the report points at the exact line.strtok_r modifies the buffer (so it must be writable, not a string literal). Note the example copies the log into a writable char log[].Logic errors.
add_distinct_port's dedup loop.NONE → you probably masked away too much or tested ACK in the wrong order. Print f in hex to see the actual bits.Questions to ask when it doesn't work: Did I check the length before indexing? Did I mask before comparing? Am I counting distinct ports or raw hits? Is my log buffer writable? What does the flags byte actually contain (print it in hex)?
This topic parses attacker-influenced data (packet bytes, log text), so memory safety is the security:
tcphdr[13] (or any offset) without first proving the buffer is long enough (n >= 14). A short or malformed packet is exactly the input an attacker would send to crash a naive parser. Out-of-bounds reads are undefined behaviour.snprintf/strncpy-with-explicit-NUL (here snprintf into e->ip) instead of strcpy, so a long or hostile IP string cannot overflow a fixed buffer.strtol can return values outside any sane range; check 1..65535 before trusting a port. Treat every field from the wire as untrusted.find_or_add returns NULL when the source table is full and the caller logs and skips rather than writing past the array. Decide what "too many sources" means rather than corrupting memory.strtok_r writes '\0' into the buffer; passing a string literal is undefined behaviour. The example uses a mutable char log[].Security & safety (defensive practice).
[ Threat model — scan detector ]
Assets: service availability, knowledge of what's exposed, the log pipeline itself
Trust boundary: the network <-- untrusted | the detector + its logs --> trusted
Entry points: raw packet bytes, the connection-log text feed
Attacker goals: map open ports unseen; or crash/flood the detector to blind it
Defensive habits, kept at least as detailed as any offensive note here: prefer prevention (firewall rate limits on new connections per source) and detection (this fan-out + flag logic) over chasing the attacker. Log enough to act, never anything sensitive: record the source IP, the distinct-port count, the timestamp, and the matched signature — but never authentication tokens, session cookies, passwords, or packet payloads. When alerting, include the threshold that fired so an analyst can tune false positives. Verify a mitigation by replaying a known-scan fixture and confirming exactly one alert fires, then replaying a busy-but-legitimate fixture and confirming zero alerts. All testing stays on localhost, containers, or an authorized lab VM.
Concrete uses.
iptables/nftables can rate-limit new connections per source IP — a direct fan-out defence — and drop packets with impossible flag combinations (NULL/XMAS), exactly the signatures from this lesson.Professional best-practice habits.
Beginner rules: validate every length and integer from untrusted input before use; mask bits, do not eyeball them; use bounded string functions; check return values of parsing functions; name constants (F_SYN, threshold) instead of scattering magic numbers.
Advanced habits: make the threshold and time window configurable per environment rather than hard-coded; track fan-out in a sliding time window so old activity ages out; combine signals with a score instead of a single hard rule to balance false positives and negatives; log structured, tunable alerts (signature, count, threshold) and never log secrets; write regression fixtures (one known scan, one busy-but-clean source) so detector changes are testable; and document the assumptions so the next defender can retune safely.
Beginner 1 — single-flag classifier.
Write int is_xmas(uint8_t flags) that returns 1 when the FIN, PSH, and URG bits are all set and ACK is not, else 0. Example: is_xmas(0x29) == 1, is_xmas(0x39) == 0 (ACK present). Constraints: mask before testing; no I/O. Hint: check (flags & F_ACK) == 0 first. Concepts: bitwise masks, flag signatures.
Beginner 2 — distinct port count.
Write int distinct_ports(const int *ports, int n) returning the number of distinct values in ports[0..n-1]. Example: {443,443,80} → 2. Constraints: handle n == 0 (return 0); do not modify the input. Hint: for each element, check whether it appeared earlier. Concepts: fan-out counting, arrays.
Intermediate 1 — threshold detector.
Write int is_port_scan(int distinct, int threshold) returning 1 if distinct >= threshold, else 0, plus a small main that prints SCAN/ok for a few pairs. Constraints: treat a negative distinct as invalid input (return -1). Hint: keep the comparison and the I/O separate so the core is testable. Concepts: thresholds, false positive/negative trade-off.
Intermediate 2 — parse and tally one source.
Given a writable log string in the format "<port>\n<port>\n..." for a single source, compute its distinct-port fan-out. Requirements: use strtok_r, validate each port is 1..65535, skip malformed lines. Example: "22\n22\n80\nbad\n443\n" → 3. Hint: reuse your distinct_ports idea while parsing. Concepts: C strings, input validation, fan-out.
Challenge — combined detector.
Write int count_scanners(const char *log, int threshold): parse a multi-source log of "<src-ip> <dst-port>" lines and return how many source IPs have a distinct-port fan-out greater than threshold. Requirements: bounded source table with graceful handling when full; reject malformed lines and out-of-range ports; copy the log into a writable buffer before tokenizing; do not crash on an empty log. Example: a source touching 11 distinct ports with threshold = 10 counts as a scanner. Hint: combine find_or_add + add_distinct_port from the lesson example. Concepts: everything above — masks (if you add a flag field), fan-out, validation, memory safety. Do not just paste the lesson code; adapt it to the exact signature.
0x02, NULL 0x00, FIN 0x01, XMAS 0x29. Mask with & before comparing — never test the raw byte with ==.0x10) set, so NULL/FIN/XMAS are easy to flag, while a lone SYN is ambiguous and needs a second signal.