Classify a TCP scan from its flag byte

Challenge

Classify a TCP probe from the flags byte of its header — SYN, NULL, FIN, and XMAS scans each leave a distinctive flag combination.

Task

Implement int tcp_scan_type(const uint8_t *tcphdr, size_t n). The TCP flags are in byte 13 (low 6 bits: FIN=0x01, SYN=0x02, RST=0x04, PSH=0x08, ACK=0x10, URG=0x20). After masking off the reserved bits, return:

• 1 — SYN scan (SYN only, 0x02)
• 2 — NULL scan (no flags, 0x00)
• 3 — FIN scan (FIN only, 0x01)
• 4 — XMAS scan (FIN+PSH+URG, 0x29)
• 0 — anything else (e.g. a normal SYN-ACK)
• -1 — tcphdr == NULL or n < 14

Input

• tcphdr: a fixed TCP-header byte buffer the grader provides (may be NULL); only byte 13 is read.
• n: the header length in bytes (must be at least 14).

Output

Returns the scan-type code 0..4, or -1 for invalid input.

Example

flags byte 0x02   ->   1   (SYN scan)
flags byte 0x00   ->   2   (NULL scan)
flags byte 0x01   ->   3   (FIN scan)
flags byte 0x29   ->   4   (XMAS scan)
flags byte 0x12   ->   0   (SYN-ACK — normal traffic)
n = 13            ->   -1  (header too short)
tcphdr = NULL     ->   -1

Edge cases

• A NULL scan is genuinely zero flags (0x00).
• Mask with 0x3F first so reserved bits don't change the classification.
• n < 14 or NULL: return -1.

Why this matters

SYN / NULL / FIN / XMAS scans each leave a tell-tale flag combination. Recognising them is how a detector classifies probe traffic.

Input format

A fixed TCP-header buffer tcphdr (may be NULL; only byte 13 read) and its length n (>= 14).

Output format

A scan-type code: 1=SYN, 2=NULL, 3=FIN, 4=XMAS, 0=other, -1 if NULL or n < 14.

Constraints

Mask the flags byte with 0x3F before matching the four signatures.

Starter code

#include <stdint.h>
#include <stddef.h>
int tcp_scan_type(const uint8_t *tcphdr, size_t n) {
    /* TODO */
    (void)tcphdr; (void)n;
    return -1;
}

Common mistakes

Not masking reserved bits. Treating SYN-ACK as a SYN scan.

Edge cases to handle

NULL scan is genuinely zero flags. Short header.

Complexity

O(1).

Background lessons

• Detecting port scans — flags and fan-out

Up next

• Detect port scanners in a connection log