Identify the file format from the first 8 bytes

Challenge

Identify an unknown file's format from its first few magic bytes — the first step of forensic triage, the same trick file(1) uses.

Task

Implement int detect_format(const unsigned char *buf, int len) that inspects the leading bytes of buf and returns a format code.

Input

• buf: a byte buffer the grader passes (the start of a file).
• len: the number of valid bytes in buf.

Output

Returns an int code based on the magic bytes:

• 1 — ELF (\x7fELF)
• 2 — Mach-O (\xfe\xed\xfa\xce, \xfe\xed\xfa\xcf, or their reversed forms — accept all four)
• 3 — Windows PE (MZ at offset 0)
• 4 — Java class file (\xca\xfe\xba\xbe)
• 5 — WebAssembly (\x00asm)
• 0 — anything else, or if buf is NULL or len < 4

Example

{0x7f,'E','L','F'}        ->   1
{0xfe,0xed,0xfa,0xce}     ->   2
{'M','Z',0x90,0}          ->   3
{0xca,0xfe,0xba,0xbe}     ->   4
{0x00,'a','s','m'}        ->   5
{0x12,0x34,0x56,0x78}     ->   0
len < 4, or NULL buf      ->   0

Edge cases

• A buffer shorter than 4 bytes returns 0.
• A NULL buffer returns 0.

Rules

• Read only buf[0..3]; never index past len.

Why this matters

Forensic triage of an unknown file always starts with magic-byte detection. file(1) does it; you'll build the same primitive in 30 lines of C.

Input format

A byte buffer buf and its valid length len.

Output format

An int code: 1=ELF, 2=Mach-O, 3=PE, 4=Java class, 5=wasm, 0=other/too short.

Constraints

Read only buf[0..3]; never index past len; NULL or len<4 returns 0.

Starter code

int detect_format(const unsigned char *buf, int len) { /* TODO */ (void)buf; (void)len; return 0; }

Common mistakes

Indexing buf[3] without checking len >= 4.

Edge cases to handle

Empty buffer; very short buffer.

Complexity

O(1).

Background lessons

• ELF header — the binary's calling card
• /proc forensics — a cookbook

Up next

• Parse a mock ELF64 header — extract version, machine, entry