Parsing server logs

Overview

Security objective. The asset you protect here is your own web server's access log and the story it tells. A log is evidence. The threat is an attacker probing your site — requesting hidden paths, hammering a login, or scanning for known-vulnerable URLs — and the defensive skill is turning a wall of raw text into countable, reviewable facts so you can detect that activity early. You are not attacking anything; you are reading records your own server already wrote.

Every mainstream web server (Apache, Nginx, and many proxies) records each HTTP request as a single line of text. The dominant shape is the Common Log Format (CLF) and its close relative the Combined format. One request, one line, a fixed field order. That regularity is exactly what makes a small C program a good tool: you can walk each line, pick out the fields you care about, and count things.

This lesson builds directly on your two prerequisites. From fgets for safe line reading you already know how to pull one line into a fixed buffer without overrunning it — that is the backbone of any log reader, because logs can be gigabytes and you never load the whole file at once. From C strings you know that a C string is a char array ending in \0, and that functions like strchr, strncmp, and sscanf operate on those arrays. Here we combine both: fgets gives us a line, string functions carve fields out of it.

Where this fits: parsing is the raw material for the next skills in this track. Once you can reliably split a line into fields, detecting failed logins (the next lesson) and rate-based alerting become simple counting problems. Parse first, decide second.

Why it matters

In real, authorized security work, log parsing is one of the highest-leverage skills you can have, and it shows up daily:

• Incident response. After a suspected breach, the first question is "what did they touch, and when?" The answer lives in access logs. A responder who can slice logs by IP, status code, and time window reconstructs the timeline in minutes instead of scrolling for hours.
• Detection engineering. Rules that page an on-call analyst ("one IP got 200 x 404s in 60 seconds") are built on top of parsed fields. If you can't extract the status code cleanly, you can't alert on it.
• Threat hunting. Proactively searching your own logs for weird patterns — odd user-agents, requests to /.git/, long query strings — depends on reliable field extraction.
• Compliance and forensics. Standards like PCI-DSS require retained, reviewable logs. Being able to demonstrate what you review and how matters in an audit.

Writing a parser yourself (rather than only using a SIEM) teaches you what the fields actually are and where they break — malformed lines, injected control characters, spoofed fields. That understanding is what separates someone who runs tools from someone who can trust and debug them. And because you're only ever reading logs from systems you own or are contracted to assess, it's a skill you can practise safely from day one.

Core concepts

1. The Common Log Format (CLF)

Definition. CLF is a plain-text, one-line-per-request format written by web servers. The field order is fixed.

Plain explanation. Think of each line as a tiny fixed-column record with no header. You always know field 1 is the IP, the bracketed chunk is the time, the quoted chunk is the request.

How it works. A canonical line:

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

When / when not. CLF (or Combined, which adds referrer and user-agent) is what you'll parse for HTTP. It is not the format for syslog, JSON logs, or firewall logs — check before assuming field positions.

Pitfall. The timestamp and the request line each contain spaces. Splitting naively on every space shreds those fields. You must treat [...] and "..." as single units.

2. Streaming a file line by line

Definition. Reading one line into a fixed buffer, processing it, then reusing the buffer for the next line.

Plain explanation. You never need the whole file in memory. fgets(buf, sizeof buf, fp) reads up to one line (or sizeof buf - 1 bytes) and null-terminates it — the safe pattern from your prerequisite.

How it works. Loop while (fgets(line, sizeof line, fp)). Each iteration you get one record. If a line is longer than the buffer, fgets returns the first chunk and the rest comes next call — something to guard against.

When / when not. Always stream logs; never fread a multi-gigabyte file into one allocation.

Pitfall. Forgetting that fgets keeps the trailing \n. Strip it or your field comparisons may fail.

3. Field extraction: strchr, strncmp, sscanf

Definition. Standard-library string tools that locate delimiters and copy or compare bytes.

Plain explanation. strchr(s, '[') finds the timestamp start; strchr(s, '"') finds the request start. sscanf can pull the two integers (status, size) after the closing quote. strncmp checks a token like " 404 " without walking off the end.

Pitfall. sscanf("%s") and strcpy are unbounded — a hostile or truncated line can overflow your destination. Always bound the copy (%NNs, strncpy + manual \0, or snprintf).

4. From fields to a defensive signal

Definition. Aggregating parsed fields into counts that indicate probing or abuse.

Plain explanation. One 404 is nothing. Two hundred 404s from one IP in a minute is a directory sweep. A cluster of 401/403 on a login path is credential guessing. Parsing gives you the fields; counting per-IP or per-status per-window gives you the signal.

Pitfall — the false positive. A broken link, a search-engine crawler, or your own uptime monitor also generate repetitive requests. A signal is a hypothesis, not a verdict; confirm with context before acting.

Threat model

            TRUST BOUNDARY (network edge)
   Internet   |                          Your infrastructure
              |
 [ Clients ]  |   [ Web server ]  --writes-->  [ access.log ]
  legit ----->|-->  Apache/Nginx                    |
  attacker -->|        |                            v
  (probes,    |        | (entry point:        [ Your log parser (C) ]
   scanners)  |        |  HTTP requests)             |
              |                                     v
              |                            [ Analyst review / alerts ]

 Assets:        access.log integrity + the truthful record it holds
 Entry points:  attacker-controlled request fields land IN the log
                (URL path, method, user-agent) — treat log CONTENT as untrusted input
 Trust boundary: the network edge; also the boundary between
                 "raw log bytes" and "your parser's buffers"

Knowledge check.

1. What asset are you protecting when you parse an access log, and what makes it valuable?
2. Where is the trust boundary between attacker-influenced data and your parser? (Hint: who controls the URL path that ends up in the log line?)
3. A spike of 404s from one IP looks like probing. Name one benign cause that would produce the same pattern (a false positive) and how you'd confirm which it is.

Syntax notes

The core loop is stream a line, then carve fields. Key building blocks:

#include <stdio.h>
#include <string.h>

char line[8192];                       /* one CLF line fits comfortably */
FILE *fp = fopen(path, "r");           /* open for reading, check for NULL */

while (fgets(line, sizeof line, fp)) { /* safe: never writes past the buffer */
    line[strcspn(line, "\n")] = '\0';  /* strip trailing newline in place   */

    char *ts  = strchr(line, '[');     /* timestamp begins at '['            */
    char *req = strchr(line, '"');     /* request line begins at first '"'   */

    /* Pull the two trailing integers (status, size) with a BOUNDED scan.
       %*[^\"] skips to the closing quote; then two ints.                    */
    int status = 0;
    long size  = 0;
    char *after = req ? strrchr(line, '"') : NULL;   /* closing quote        */
    if (after && sscanf(after + 1, " %d %ld", &status, &size) >= 1) {
        /* status now holds the HTTP code, e.g. 200 or 404 */
    }
}

Annotations:

• strcspn(line, "\n") returns the index of the first newline (or end); assigning '\0' there trims it without a separate strlen.
• strchr / strrchr return a pointer into line or NULL. Always check for NULL before dereferencing.
• In sscanf, always bound %s conversions (e.g. %255s) and prefer %d/%ld for numbers. Never use bare %s into a fixed buffer.
• sizeof line (not a hard-coded number) keeps the buffer size and the fgets limit in sync.

Lesson

The Common Log Format

Most web servers write access logs in the Common Log Format (CLF). Each request becomes one line. Here is an example:

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

Fields in each line

A single line packs several fields together, in this order:

• IP — the client address (127.0.0.1)
• Identifier — the RFC 1413 identity, usually - (not provided)
• User — the authenticated user (frank)
• Timestamp — the request time, in brackets ([10/Oct/2000:13:55:36 -0700])
• Request line — the HTTP request, in quotes ("GET /apache_pb.gif HTTP/1.0")
• Status — the HTTP response code (200)
• Size — the response size in bytes (2326)

Parsing in C

A small C parser can pull these fields out one line at a time:

• Read each line with fgets.
• Split the line character by character, watching for the brackets and quotes that group multi-word fields.

This is useful for incident analysis — but only on logs from your own server.

Code examples

Below: an insecure parser, the secure version, and checks that prove the fix. All examples run on a local file you create yourself — no network, no live server.

(1) WARNING: intentionally vulnerable — use only in a local, isolated, authorized lab. Do not deploy.

/* insecure_parse.c — DO NOT DEPLOY. Demonstrates the overflow to AVOID. */
#include <stdio.h>

int main(void) {
    FILE *fp = fopen("access.log", "r");
    char line[8192];
    char ip[16];                 /* far too small, and no bound below */
    while (fgets(line, sizeof line, fp)) {   /* fp not checked for NULL! */
        sscanf(line, "%s", ip);  /* UNBOUNDED: a long IP field overflows ip[] */
        printf("ip=%s\n", ip);
    }
    return 0;   /* fp never fclose'd */
}

Why it's dangerous: sscanf("%s") writes as many bytes as the field has, with no limit — a crafted or corrupted log line overruns ip[16] (a stack buffer overflow, undefined behaviour). fp is used without a NULL check, so a missing file dereferences NULL. The file is never closed. Log content is attacker-influenced, so this is a real exposure, not a theoretical one.

(2) SECURE version

/* secure_parse.c — bounded, checked, cleaned up. C11. */
#include <stdio.h>
#include <string.h>

/* Extract fields from one CLF line into out-params.
   Returns 1 on a usable parse, 0 otherwise. Never writes past a buffer. */
static int parse_clf(const char *line, char *ip, size_t ip_sz, int *status) {
    /* Bounded copy of the leading IP token (stop at first space). */
    size_t i = 0;
    while (line[i] && line[i] != ' ' && i + 1 < ip_sz) { ip[i] = line[i]; i++; }
    ip[i] = '\0';
    if (i == 0) return 0;                       /* no IP -> unusable */

    /* Status is the integer right after the CLOSING quote of the request. */
    const char *close = strrchr(line, '"');
    if (!close) return 0;                        /* malformed: no request line */
    if (sscanf(close + 1, " %d", status) != 1) return 0;
    if (*status < 100 || *status > 599) return 0; /* not a real HTTP code */
    return 1;
}

int main(void) {
    FILE *fp = fopen("access.log", "r");
    if (!fp) { perror("fopen access.log"); return 1; }   /* checked */

    char line[8192];
    long total = 0, not_found = 0;
    while (fgets(line, sizeof line, fp)) {
        line[strcspn(line, "\n")] = '\0';
        char ip[64];
        int status;
        if (!parse_clf(line, ip, sizeof ip, &status)) continue; /* skip junk */
        total++;
        if (status == 404) {
            not_found++;
            /* Defensive signal: log the observation (see Security notes). */
            printf("NOTFOUND src=%s status=%d\n", ip, status);
        }
    }
    if (ferror(fp)) { perror("read"); fclose(fp); return 1; }
    fclose(fp);                                            /* cleanup */

    printf("summary: %ld requests, %ld were 404\n", total, not_found);
    return 0;
}

(3) VERIFY — prove the fix rejects bad input and accepts good input

Create a small fixture and run it. This is safe: you author the file yourself.

# Build both
cc -std=c11 -Wall -Wextra -fsanitize=address secure_parse.c -o secure_parse

# GOOD line (real 404) + BAD lines (malformed, oversized IP, no quotes)
printf '%s\n' \
  '127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /admin HTTP/1.0" 404 12' \
  '10.0.0.5 - - [10/Oct/2000:13:55:37 -0700] "GET /ok HTTP/1.0" 200 99' \
  'this-line-has-no-quotes-and-no-status' \
  'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA no-quote 404' \
  > access.log

./secure_parse

Expected behaviour: the two well-formed lines are counted (summary: 2 requests), the /admin 404 is flagged (NOTFOUND src=127.0.0.1 status=404), and the two malformed lines are skipped rather than crashing or overflowing. Running the insecure version against the oversized-IP line under -fsanitize=address instead reports a stack-buffer-overflow — that contrast is the proof that bounding the copy fixed a real bug. Verifying that bad input is rejected and good input is accepted is the mitigation-verification step; a parser that never rejects anything is not actually validating.

Line by line

Walking the SECURE parse_clf on the line: 127.0.0.1 - - [.../-0700] "GET /admin HTTP/1.0" 404 12

1. Bounded IP copy. The while loop copies characters until it hits a space or fills the buffer (i + 1 < ip_sz leaves room for the terminator). It stops at the space after 127.0.0.1.
2. Terminate + sanity check. ip[i] = '\0' closes the string. If i == 0 (line started with a space) we bail — no usable IP.
3. Find the request end. strrchr(line, '"') returns a pointer to the last ", i.e. the closing quote of "GET /admin HTTP/1.0". Using the last quote is what lets the status parse start in the right place even though the URL could contain characters we don't model.
4. Guard the pointer. If strrchr returned NULL (no quote at all), we return 0 — this is how the this-line-has-no-quotes fixture line is rejected.
5. Parse the status. sscanf(close + 1, " %d", status) reads past the quote and any spaces, capturing 404. The leading space in " %d" skips whitespace. If it can't read an integer, we return 0.
6. Range-validate. 100..599 is the only valid HTTP status range; anything else means we mis-parsed, so we reject. This is validation, not decoration.

Trace of key values for two lines:

In main, each returned-1 line increments total; a status == 404 also increments not_found and prints a one-line observation. The oversized-IP line is safely truncated by the bound and then rejected for lacking a quote — no overflow, exactly the outcome the insecure version failed to guarantee.

Common mistakes

1. Splitting on every space.

• WRONG: sscanf(line, "%s %s %s %s", a,b,c,d) to grab "the fields".
• WHY: the timestamp and request line contain spaces, so the fields shift and everything after them is wrong.
• CORRECTED: anchor on the structural delimiters — [ for time, " for request, and the closing " for the trailing integers.
• RECOGNISE/PREVENT: if your status column sometimes shows a date fragment or "HTTP/1.0", you split wrong. Test against a line with a multi-word field.

2. Unbounded copies.

• WRONG: sscanf(line, "%s", ip); or strcpy(ip, field);.
• WHY: log content is attacker-influenced; an oversized field overflows the destination (UB, potential crash or worse).
• CORRECTED: bound every copy (%63s, or a manual loop with i + 1 < size, or snprintf).
• RECOGNISE/PREVENT: build with -fsanitize=address -Wall -Wextra; it flags the overflow immediately.

3. Not checking fopen.

• WRONG: using fp right after fopen with no NULL test.
• WHY: a missing/permission-denied file returns NULL; fgets(line, n, NULL) is undefined.
• CORRECTED: if (!fp) { perror(...); return 1; }.

4. Treating a signal as a verdict.

• WRONG: "100 x 404 from one IP, therefore it's an attacker — block them."
• WHY: crawlers, broken deploys, and monitors produce the same pattern (false positive), and the IP may be a shared proxy.
• CORRECTED: treat counts as a hypothesis; corroborate with user-agent, timing, and target paths before acting, and record your reasoning.

5. Trusting the client IP as identity.

• WRONG: assuming the leading IP uniquely identifies a person.
• WHY: proxies, NAT, and load balancers mean many users share one IP, and headers like X-Forwarded-For can be spoofed.
• CORRECTED: treat the IP as one weak signal among several; never make an authorization decision on it alone.

Debugging tips

Common errors and how to chase them:

• Status column is garbage. You probably used the first quote instead of the last. Print close and the substring after it. Switch strchr to strrchr for the closing quote.
• Every line is skipped (total: 0). Check whether fgets is even reading — print the first line. A missing newline strip or a wrong path (fopen returned NULL, but you didn't check) are the usual causes. perror tells you which.
• Crash / ASan report on some inputs. Build with cc -std=c11 -Wall -Wextra -fsanitize=address,undefined. ASan pinpoints the exact buffer and line of an overflow; UBSan catches bad integer/pointer ops. Re-run against your malformed fixture.
• Counts look too high. You may be double-counting continuation lines: a line longer than the buffer is delivered to fgets in two calls. Detect it — if the buffer is full and the last char isn't \n, you're mid-line; drain to the next newline before treating it as a new record.
• Locale/encoding surprises. Logs can contain non-ASCII or injected control bytes in the URL. Don't printf raw log fields to a terminal unfiltered (they can contain escape sequences); sanitize or hex-dump suspicious values.

Questions to ask when it fails:

1. Did fopen succeed? (Always check first.)
2. Is the field I'm reading grouped by spaces, brackets, or quotes — and am I anchoring on the right delimiter?
3. Is every copy bounded by the destination size?
4. Does my parser reject a deliberately malformed line, or does it silently produce wrong numbers?

Memory safety

Security & safety

Memory safety (C). Every field you extract from a log line comes from untrusted content — the URL path, method, and user-agent are attacker-controlled and land verbatim in the line. Treat the whole line as hostile input:

• Bound every copy to the destination size; never strcpy/bare %s into a fixed buffer.
• Check fopen, check fgets's return, check sscanf's return count before trusting out-params.
• Always fclose; check ferror after the loop.
• Build and test with -fsanitize=address,undefined -Wall -Wextra.

Detection & logging — what to record. When your parser flags something, write a structured, one-line observation with:

• Timestamp (from the log line and/or your analysis run).
• Source — the client IP / identifier (a weak but useful signal).
• Resource — the requested path and method.
• Result — the HTTP status (e.g. 404, 401).
• Security decision — what you concluded or the rule that fired (e.g. rule=404-sweep count=213 window=60s).
• Correlation id — a run/session id so related findings can be joined later.

What to NEVER log. Do not write passwords, session cookies, Authorization headers or bearer tokens, API keys, private keys, full payment card numbers (PANs), or unneeded personal data. These often appear inside query strings or headers in raw traffic — if you pull request fields, redact secrets before writing your own output, or you turn your analysis log into a new leak.

Events that signal abuse. Bursts of 404 to non-existent paths (directory/vuln scanning), repeated 401/403 on auth endpoints (credential guessing), one IP requesting many distinct sensitive paths (/.git/, /admin, /wp-login.php), and unusually long or encoded URLs.

How false positives arise. Search-engine crawlers, uptime/health monitors, a freshly broken deploy (mass 404 from your own users), CDN or proxy IPs aggregating many real users, and security scanners you scheduled. A good detection records enough context (user-agent, timing, target set) that a human can dismiss a false positive quickly — and every rule needs a benign-cause review before it pages anyone.

Real-world uses

Authorized use case. You run a small web app and want an early warning for path-probing. You copy your own access.log to a workstation (or read it in place), run your parser to count 404s per source IP over rolling windows, and print any IP above a threshold for a human to review. Everything is on infrastructure you own; nothing is sent anywhere.

Professional best-practice habits:

• Input validation. Treat log content as untrusted; bound copies; range-check parsed numbers; reject rather than guess on malformed lines.
• Least privilege. Run the parser as a non-root user with read-only access to the log. It never needs write access to the server or network access.
• Secure defaults. Default to reporting, not blocking. Automated blocking on a raw parser is how you take yourself offline with a false positive.
• Logging & error handling. Emit structured, redacted findings with a correlation id; check every library return; fail loudly (perror) rather than silently.

Beginner vs advanced:

Practice tasks

All tasks are lab-only: work on log files you create yourself or on logs from a system you own. Each ends by remediating and verifying, not attacking.

Authorization checklist (before any lab): (1) You own the system or have written authorization and scope. (2) The lab is local — your own file, localhost, or an isolated VM/container. (3) No third-party or production data. (4) You have a cleanup/reset step.

Beginner 1 — Field extractor.

• Objective: parse one CLF line into IP, status, and size.
• Requirements: read from a file with fgets; use bounded copies; print ip=... status=... size=....
• Input/Output: input a small access.log you author; output one summary line per record.
• Constraints: no strcpy, no bare %s; must compile with -Wall -Wextra.
• Hints: anchor status parsing on the last ".
• Concepts: fgets, strrchr, bounded sscanf.

Beginner 2 — 404 counter.

• Objective: count how many lines have status 404.
• Requirements: skip malformed lines; print the total.
• Input/Output: a fixture mixing 200s, 404s, and junk lines; output the count.
• Constraints: reject any line that fails validation instead of counting it.
• Hints: reuse your extractor; range-check the status.
• Concepts: validation, counting, control flow.

Intermediate 1 — Per-IP 404 tally.

• Objective: report, per source IP, how many 404s it caused.
• Requirements: a simple table (fixed-size array of {ip, count} is fine); print IPs sorted by count descending.
• Input/Output: fixture with several IPs; output count ip lines.
• Constraints: bounded IP storage; handle an IP appearing many times.
• Hints: linear scan of a small table is acceptable for the lab.
• Concepts: aggregation, arrays of structs.

Intermediate 2 — Windowed spike detector.

• Objective: flag any IP with more than N 404s within a 60-second window.
• Requirements: parse the bracketed timestamp enough to compare times; print a finding line per flagged IP.
• Input/Output: fixture with a burst and some noise; output findings with count and window.
• Constraints: findings must include a benign-cause note field (e.g. possible=crawler?).
• Hints: you can compare timestamps as strings if the format is fixed, or convert with strptime.
• Concepts: time handling, thresholds, false-positive awareness.
• Defensive conclusion: for each finding, write the remediation you'd recommend (e.g. rate-limit the path, add auth, fix the broken link) and verify by re-running against a fixture where the benign cause is present to confirm you don't over-flag.

Challenge — Robust, redacting parser.

• Objective: a parser that survives hostile input and never leaks secrets.
• Requirements: handle over-long lines (drain to newline), injected control bytes (sanitize before printing), and Combined-format lines (extra quoted fields); redact anything that looks like a token/cookie/key in fields you echo; emit structured findings with a correlation id.
• Constraints: build clean under -fsanitize=address,undefined -Wall -Wextra; must reject malformed input without crashing.
• Hints: write a fixture that tries to break you (huge IP, embedded \n-escapes, a fake Authorization: Bearer ... in the URL) and confirm each is handled.
• Concepts: defensive parsing, redaction, sanitization, mitigation verification.
• Defensive conclusion + cleanup/reset: delete your test fixtures (rm access.log), and confirm no captured log data or findings containing real IPs/PII persist outside the lab.

Summary

Main concepts. Web servers write access logs in the Common Log Format: one request per line, fixed field order (IP, ident, user, [timestamp], "request", status, size). The timestamp and request line contain spaces, so you must anchor on [, ", and the closing " rather than splitting on whitespace. Stream the file with fgets — never load it whole — and treat every field as untrusted input because attackers control what goes into the URL, method, and headers that end up in the line.

Key syntax/commands. fgets(line, sizeof line, fp) to read; line[strcspn(line,"\n")]='\0' to trim; strrchr(line,'"') to find the request end; bounded sscanf(close+1, " %d", &status) for the status; range-check 100..599. Build and test with cc -std=c11 -Wall -Wextra -fsanitize=address,undefined.

Common mistakes. Splitting on every space; unbounded copies (strcpy, bare %s) that overflow; skipping the fopen/fgets/sscanf return checks; treating a 404 spike as a verdict rather than a hypothesis; trusting the client IP as identity.

What to remember. Parse first, decide second. Bound every copy. Reject malformed lines instead of guessing. Log findings with timestamp, source, resource, result, decision, and a correlation id — and never log passwords, tokens, cookies, keys, full card numbers, or unneeded PII. Only ever analyse logs from systems you own or are authorized to review, and confirm a mitigation works by testing that it rejects bad input and accepts good input.

Practice with these exercises

• Count 404 responses
• Count distinct client IPs