Log parsing — line iteration + classification

Overview

A log is the running diary of a program. Every time something noteworthy happens — a request arrives, a login fails, a disk fills up — the program appends a line of text describing it. Over hours or days these lines pile into a file (or a stream) that is the historical record of what the system actually did.

Log parsing is the act of reading that record back and turning it into answers: How many errors happened today? Which IP tried to log in the most? When did the failures start? This is one of the most common everyday tasks in operations, web analytics, billing, and security work — from incident response to penetration testing.

This lesson builds directly on two things you already know. From C strings you have strstr, strcspn, strlen, and the idea that a C string is a NUL-terminated array of char. From fopen and the stdio model you have fopen, the FILE * handle, and the buffered read/write model. Log parsing is essentially those two skills combined into one tight loop.

The recipe almost never changes:

1. Open the file.
2. Loop over it with fgets, one line per iteration.
3. Clean the line (strip the newline).
4. Classify the line (which category does it belong to?).
5. Aggregate (bump a counter, or update a running statistic).
6. At end of stream, summarise.

The term stream-processing describes steps 2–6: you process each record as it arrives and never hold more than one line in memory at a time. That property is what lets the same 40-line program handle a 2 KB file or a 20 GB file without changing.

Why it matters

Logs are the only record of what a system did when nobody was watching. They hold the forensic evidence of an incident, the early-warning markers of an anomaly, and the raw numbers behind dashboards and bills.

Reading them quickly and correctly is a daily skill for many roles:

• Operations / SRE: spotting the spike of ERROR lines that preceded an outage.
• Security: detecting a burst of failed logins that signals a brute-force attempt (exactly what tools like fail2ban watch for).
• Product / billing: counting API calls per customer to drive usage-based invoices.
• Forensics / IR: reconstructing a timeline from timestamps after a breach.

Getting it wrong is also a real security problem. If you log a value that came from outside your program — a username, a filename, a User-Agent header — without cleaning it first, an attacker can embed a fake newline and forge whole log entries. That is log injection (CWE-117), and it can hide an intrusion or frame an innocent user. So a log parser is not just a counting tool; it sits on a trust boundary, and writing one teaches you to treat text as untrusted by default.

Core concepts

1. Stream, don't slurp

Definition. Streaming means reading and processing the input incrementally — here, one line per loop iteration — rather than reading the entire file into a buffer first ("slurping").

Why it matters. Logs grow without bound. A web server can produce gigabytes per day. If you malloc space for the whole file, a large (or maliciously large) log can exhaust memory and crash the process. Streaming uses a fixed, tiny amount of memory no matter how big the file is.

How it works internally. fgets copies bytes from the FILE *'s internal buffer into your small fixed buffer, stopping at a newline or when your buffer is nearly full. The standard library refills its buffer from disk behind the scenes, so your loop only ever sees one line at a time.

Disk file (huge)        stdio buffer (libc)      your buffer (4 KB)
+----------------+      +---------------+        +-------------+
| line 1\n       |  -->  | line 1\n      |  fgets  | line 1\0    |
| line 2\n       |      | line 2\n ...  | ----->  +-------------+
| ... millions  |      +---------------+         (reused each
+----------------+                                 iteration)

When NOT to stream: if you genuinely need random access or multiple passes over a small, known-bounded file, reading it once into memory can be simpler. For logs, default to streaming.

Pitfall: forgetting that each fgets overwrites your buffer. If you need to keep a line, copy it out before the next call.

Knowledge check: Your buffer is char line[4096]. A single log line in the file is 10,000 bytes long. What does the first fgets return, and what is left for the next call?

2. Strip the trailing newline

Definition. fgets stores the newline character '\n' as part of the line (if it fits). Stripping means replacing that '\n' with a NUL terminator so the string ends cleanly.

Why it matters. String comparisons include the newline. strcmp(line, "ERROR") will fail on "ERROR\n", and printing the line adds a stray blank line. Most comparison bugs in beginner parsers come from the un-stripped newline.

How it works. strcspn(line, "\n") returns the index of the first newline (or the length of the string if there is none). Writing 0 ('\0') at that index cuts the string off right before the newline.

before:  E R R O R \n \0
                  ^ strcspn returns 5
after:   E R R O R \0 \0

Pitfall: the very last line of a file often has no trailing newline. strcspn handles this correctly (it returns the full length and you overwrite the existing '\0' harmlessly), which is exactly why it is safer than blindly doing line[strlen(line)-1] = 0.

Knowledge check: Why is line[strlen(line) - 1] = '\0'; dangerous on a possibly-empty line, and how does strcspn avoid that bug?

3. Cap the line length

Definition. The second argument to fgets is the maximum number of bytes (including the NUL) it will write. That fixed cap is your defence against oversized input.

Why it matters. An attacker who controls part of a log line could try to send a line megabytes long to stress your parser. With a fixed buffer, fgets simply reads at most sizeof buf - 1 bytes and leaves the rest for the next call — your memory use stays flat.

Structure / habit. Always pass sizeof buffer (not a hardcoded number) so the cap and the buffer can never drift apart:

char line[4096];
while (fgets(line, sizeof line, fp)) { /* ... */ }

Pitfall: a line longer than the buffer is split across multiple fgets calls. If your classifier looks for a marker near the start that is fine; if you need the whole line, detect the split (no '\n' was read) and decide whether to skip the oversized record or keep reading it.

4. Classify with substring search

Definition. Classification is deciding which bucket a line belongs to. The simplest method is testing whether a marker string appears in the line with strstr(line, marker).

How it works. strstr returns a pointer to the first occurrence of the needle inside the haystack, or NULL if it is absent. You treat that pointer as a boolean ("found / not found") or use it as the starting point to extract a field.

line:   "... Failed password for root from 10.0.0.9 port 22"
                ^strstr(line,"Failed password") != NULL  -> a failed login

When to use / when not. Substring search is perfect for stable markers (ERROR, Failed password). It is not reliable for structured extraction where position matters or markers can appear inside other text — for that, parse fields explicitly (the next lesson, Parsing server logs, goes deeper).

Pitfall: substring matches anywhere, so strstr(line, "ERROR") also matches "NO_ERRORS_TODAY". Choose distinctive markers, or anchor on a delimiter.

5. Aggregate

Definition. Aggregation is folding many lines into a few numbers: counters per level, a max streak, a per-key tally.

Structure. Keep one accumulator per category and update it inside the loop. After the loop ends, the accumulators hold the answer.

  for each line:
      if INFO  -> info++
      if WARN  -> warn++
      if ERROR -> err++
  end:  print info, warn, err

Knowledge check: A log has 100 lines: 60 contain INFO, 30 contain WARN, 10 contain ERROR, and none contains two levels. After the loop, what are info, warn, err? Now suppose 5 lines contained both WARN and ERROR — would your three independent if checks still sum to 100?

6. Log content is untrusted (log injection)

Definition. Log injection (CWE-117) is when attacker-controlled text containing \r or \n is written into a log, creating forged extra lines that look like real entries.

Why it matters. If your app logs "login failed for user: " + username and the attacker's "username" is bob\nlogin OK for user: admin, your log now shows a fake successful admin login. That can hide an attack or frame someone.

Defensive habit. Any field that came from outside the program must have its control bytes stripped or escaped before it is logged. When reading logs, be aware that embedded control characters and NUL bytes may be present and adversarial.

Trust boundary (when WRITING logs):

  user input ──► [ sanitise: strip \r \n ] ──► log file
  (untrusted)        ^ neutralise here        (trusted record)

Pitfall: sanitising only \n but not \r. A lone \r can still confuse log viewers; strip both.

Syntax notes

The canonical streaming loop:

FILE *fp = fopen(path, "r");      /* open for reading            */
if (!fp) { perror(path); return 1; }  /* always check the result */

char line[4096];                 /* fixed cap = DoS protection   */
while (fgets(line, sizeof line, fp)) {   /* one line per pass     */
    line[strcspn(line, "\n")] = '\0';    /* strip trailing newline */
    if (strstr(line, "ERROR")) {         /* classify by marker     */
        /* aggregate: bump a counter, etc. */
    }
}
fclose(fp);                      /* release the handle           */

Key points:

• fgets returns the buffer pointer on success and NULL at end-of-file or on error — that is what ends the loop.
• sizeof line is the byte cap including the NUL, so fgets never overruns line.
• strcspn(line, "\n") gives the index of the first '\n'; writing '\0' there trims it. It is safe even when there is no newline.
• strstr returns NULL when the marker is absent, so it doubles as a found/not-found test.

Lesson

Logs are line-delimited text: each entry sits on its own line.

A parser walks the file one line at a time. For each line it does two things:

• Classify the line — for example by level (info, warn, error), or by IP, user, or route.
• Aggregate — update a counter or a running statistic.

Code examples

#include <stdio.h>
#include <string.h>

#define LINE_CAP 4096   /* refuse log lines longer than this */

/* Copy the IP that appears between "from " and " port" into out[cap].
   Returns 1 on success, 0 if the markers are missing or it would not fit. */
static int extract_ip(const char *line, char *out, size_t cap) {
    const char *from = strstr(line, "from ");
    if (!from) return 0;
    from += 5;                          /* skip past "from "            */
    const char *port = strstr(from, " port");
    if (!port) return 0;
    size_t n = (size_t)(port - from);   /* length of the IP substring   */
    if (n >= cap) return 0;             /* leave room for the NUL byte  */
    memcpy(out, from, n);
    out[n] = '\0';
    return 1;
}

int main(int argc, char **argv) {
    if (argc != 2) {
        fprintf(stderr, "usage: %s <logfile>\n", argv[0]);
        return 1;
    }

    FILE *fp = fopen(argv[1], "r");
    if (!fp) { perror(argv[1]); return 1; }  /* report why open failed  */

    long info = 0, warn = 0, err = 0, failed_logins = 0, total = 0;
    char line[LINE_CAP];
    char ip[64];

    while (fgets(line, sizeof line, fp)) {
        line[strcspn(line, "\n")] = '\0';    /* strip the newline       */
        total++;

        if (strstr(line, "INFO"))  info++;
        if (strstr(line, "WARN"))  warn++;
        if (strstr(line, "ERROR")) err++;

        if (strstr(line, "Failed password")) {
            failed_logins++;
            if (extract_ip(line, ip, sizeof ip))
                printf("  failed login from %s\n", ip);
        }
    }

    if (ferror(fp)) {            /* distinguish a read error from EOF   */
        perror("read error");
        fclose(fp);
        return 1;
    }
    fclose(fp);                 /* release the file handle             */

    printf("\nlines=%ld  INFO=%ld  WARN=%ld  ERROR=%ld  failed_logins=%ld\n",
           total, info, warn, err, failed_logins);
    return 0;
}

What it does. The program takes a log file path on the command line, streams it line by line, strips each newline, classifies lines by level marker, and counts failed SSH-style logins — printing the source IP of each one. At the end it prints a summary.

Expected output for an input like:

May 10 09:00:01 host sshd: INFO accepted login for alice
May 10 09:00:05 host sshd: Failed password for root from 10.0.0.9 port 22
May 10 09:00:07 host app: WARN disk usage 85%
May 10 09:00:09 host app: ERROR cannot reach database

the program prints:

  failed login from 10.0.0.9

lines=4  INFO=1  WARN=1  ERROR=1  failed_logins=1

Key edge cases. An empty file prints all zeros. A line with no trailing newline (the last line) is handled by strcspn. A Failed password line missing the port marker is counted but its IP is not printed (extract_ip returns 0). A line longer than LINE_CAP is split across iterations, which would inflate total; for strict counting you would detect the missing '\n' and merge the pieces.

Line by line

We trace the program on the 4-line sample above.

Setup. argc is 2, so the usage check passes. fopen(argv[1], "r") returns a valid FILE *; the !fp check is skipped. All counters start at 0.

Iteration 1 — fgets reads "...: INFO accepted login for alice\n".

• strcspn(line, "\n") finds the newline index; line[idx] = '\0' removes it.
• total becomes 1.
• strstr(line, "INFO") is non-NULL → info becomes 1. The WARN, ERROR, and Failed password searches return NULL.

Iteration 2 — fgets reads the Failed password ... from 10.0.0.9 port 22 line.

• Newline stripped; total becomes 2.
• Level searches all miss.
• strstr(line, "Failed password") matches → failed_logins becomes 1, then extract_ip runs:

Returns 1, so the program prints failed login from 10.0.0.9.

Iteration 3 — the WARN line: total→3, warn→1.

Iteration 4 — the ERROR line: total→4, err→1.

End of stream. fgets returns NULL. ferror(fp) is false (clean EOF), so we skip the error branch, fclose the file, and print the summary line: lines=4 INFO=1 WARN=1 ERROR=1 failed_logins=1.

The important mental model: at every moment, only one line exists in line[], and the counters carry the running totals across iterations.

Common mistakes

Mistake 1 — slurping the whole file.

/* WRONG: allocate the whole file, then parse */
fseek(fp, 0, SEEK_END);
long size = ftell(fp);
char *buf = malloc(size);   /* size could be gigabytes */
fread(buf, 1, size, fp);

Why it is wrong: memory use scales with file size, so a huge or hostile log can exhaust RAM. Fix: stream with fgets into a fixed buffer. Recognise it when memory blows up on big inputs but works on small test files.

Mistake 2 — comparing without stripping the newline.

if (strcmp(line, "ERROR") == 0) err++;   /* never matches "ERROR\n" */

Why it is wrong: fgets keeps the '\n', so the strings differ. Fix: line[strcspn(line, "\n")] = '\0'; first — or use strstr for substring tests. Recognise it when counters stay at 0 even though the marker is clearly present.

Mistake 3 — line[strlen(line) - 1] = '\0'; to strip the newline.

Why it is wrong: on an empty string strlen is 0 and the index -1 underflows to a huge size_t, corrupting memory; and if the last line has no newline it deletes a real character. Fix: use strcspn which is safe in both cases.

Mistake 4 — trusting ftell/fgets return without checking fopen.

FILE *fp = fopen(path, "r");
while (fgets(line, sizeof line, fp)) { ... }  /* fp may be NULL! */

Why it is wrong: if the file is missing, fp is NULL and fgets(NULL, ...) is undefined behaviour (typically a crash). Fix: check if (!fp) { perror(path); return 1; }.

Mistake 5 — logging untrusted text verbatim (log injection).

fprintf(logfp, "login failed for user: %s\n", username); /* username from user */

Why it is wrong: if username contains \n, the attacker forges extra log lines (CWE-117). Fix: strip or escape \r and \n from username before logging. Recognise it when log lines appear that no code path could have produced.

Debugging tips

Compiler errors

• implicit declaration of strstr/strcspn: you forgot #include <string.h>.
• implicit declaration of fopen/fgets: add #include <stdio.h>.
• format %ld expects long: match the printf specifier to the counter type (long ↔ %ld, size_t ↔ %zu).

Runtime errors

• Segfault immediately on a missing file: you used fp without checking it for NULL.
• Segfault inside extract_ip: the markers were not found, so a pointer is wrong; verify both strstr results are non-NULL before subtracting them.
• Garbage in the extracted IP: you forgot out[n] = '\0';, so the field is not terminated.

Logic errors

• All counts are 0: you compared with strcmp against a newline-terminated line — strip the newline or use strstr.
• Counts too high: your marker matches inside other words ("ERROR" inside "NO_ERRORS"), or long lines are being split across fgets calls and counted twice.
• Off-by-one in IP length: check whether you skipped exactly strlen("from ") (5) bytes.

Concrete steps

1. Print each line with visible delimiters: printf("[%s]\n", line); to see stray \n or \r.
2. Test against the awkward inputs below before trusting the parser.
3. Run under a sanitizer: gcc -fsanitize=address,undefined -g logparse.c catches buffer and pointer bugs immediately.

Questions to ask when it doesn't work

• Did fopen actually succeed?
• Is the newline really stripped (print with brackets)?
• Does my marker appear inside other words?
• Are any lines longer than my buffer?

Memory safety

This is a C track lesson, so the safety concerns are about buffers, pointers, and untrusted bytes.

fgets is the safe reader. Unlike gets (removed from C11 — never use it), fgets takes a size and always NUL-terminates the buffer. The size argument includes the NUL byte, so fgets(buf, sizeof buf, fp) writes at most sizeof buf - 1 data bytes plus the terminator. Pass sizeof buf, never a separate hardcoded length that could drift.

Bounds in field extraction. When copying a sub-field out of a line, always check that it fits before copying and always leave room for the terminator. In extract_ip the guard is if (n >= cap) return 0; followed by out[n] = '\0';. Off-by-one here is the classic buffer overflow.

Pointer lifetime. strstr returns a pointer into line. That pointer is only valid until the next fgets overwrites the buffer. Never stash a strstr result and use it after the next iteration — copy the bytes out first.

Initialisation. Counters must start at a known value (0). Reading an uninitialised counter is undefined behaviour and gives garbage totals.

Integer/overflow care. port - from is a pointer difference; cast it to size_t only after you know port > from (guaranteed here because port came from strstr(from, ...)).

Untrusted bytes. Log lines can contain embedded NUL bytes; everything after a NUL looks empty to string functions. Lines can also carry \r/\n from log-injection attempts. Treat the content as adversarial: cap the length, and when writing logs, neutralise control characters first.

Real-world uses

Concrete use case — brute-force detection. A small daemon streams /var/log/auth.log, counts consecutive Failed password lines per source IP, and blocks any IP that crosses a threshold. This is the core of tools like fail2ban. The same streaming-classify-aggregate loop you wrote here is the engine inside it.

Other everyday uses:

• Web-server analytics: counting requests per route or status code from an access log.
• Billing aggregation: tallying API calls per customer key.
• Anomaly alerts: firing when ERROR lines per minute exceed a baseline.
• Incident-response timelines: extracting timestamps to reconstruct what happened when.

Professional best-practice habits

Beginner rules:

• Always check fopen for NULL and fclose what you open.
• Always strip the newline before comparing.
• Pass sizeof buffer to fgets, never a literal.
• Use clear names (failed_logins, not n).

Advanced habits:

• Distinguish clean EOF from a read error with ferror before reporting success.
• Choose distinctive, anchored markers so substring matches do not over-count.
• Decide a deliberate policy for over-long lines (skip, truncate, or reassemble) rather than letting them silently inflate counts.
• When the format is structured, parse fields explicitly instead of strstr guessing — the next lesson, Parsing server logs, covers this.
• Treat all logged external input as untrusted: sanitise control characters to prevent log injection (CWE-117), and never log secrets such as passwords or tokens.

Practice tasks

Beginner 1 — Count total lines. Objective: print how many lines a log file contains. Requirements: take the path on the command line, stream with fgets, increment a counter per line, print the total. Constraint: never load the whole file into memory. Hint: the loop body can be a single count++. Concepts: streaming, the fgets loop.

Beginner 2 — Count lines at a given level. Objective: count lines containing a level string passed as the second argument. Example: ./count app.log ERROR prints ERROR: 7. Requirements: strip the newline, use strstr(line, level) to test each line. Constraint: a missing argument should print a usage message and exit non-zero. Hint: strstr returning non-NULL means "found". Concepts: classification, command-line arguments.

Intermediate 1 — Per-level summary in one pass. Objective: in a single pass, count INFO, WARN, and ERROR lines and print all three. Requirements: three independent counters updated in the loop; print a summary at end of stream. Input/output example: a 4-line file prints INFO=1 WARN=1 ERROR=1. Constraint: one pass over the file only. Hint: three independent if checks, not else if, unless you know a line can hold only one level. Concepts: aggregation, multiple counters.

Intermediate 2 — Extract and print source IPs. Objective: for every Failed password line, print the IP between from and port. Requirements: implement a bounded extract_ip like the lesson's; skip lines where either marker is missing. Constraint: never write past the output buffer; always NUL-terminate. Hint: compute the length as the pointer difference and check it against the capacity before memcpy. Concepts: substring search, pointer arithmetic, bounded copies.

Challenge — Top-N noisy IPs. Objective: read an access log, tally how many times each distinct source IP appears, and print the IPs that appear at least threshold times. Requirements: maintain a small dynamic table of (ip, count) pairs; for each line, extract the IP, look it up, and increment or insert. Input/output example: with threshold=3, print every IP seen 3+ times with its count. Constraints: stream the file (one line in memory at a time); cap the line length; handle lines with no extractable IP gracefully. Hints: a linear-scan array is fine for a first version; sanitise the extracted IP before printing it (control bytes!). Concepts: streaming, extraction, aggregation into a keyed table, untrusted input.

Summary

A log parser is a tight loop: open → stream → strip → classify → aggregate → summarise.

The most important syntax to remember:

FILE *fp = fopen(path, "r");
if (!fp) { perror(path); return 1; }
char line[4096];
while (fgets(line, sizeof line, fp)) {
    line[strcspn(line, "\n")] = '\0';   /* strip newline */
    if (strstr(line, "ERROR")) err++;   /* classify + aggregate */
}
fclose(fp);

The common mistakes: slurping the whole file, comparing without stripping the newline, strlen(line)-1 on an empty line, not checking fopen, and logging untrusted text verbatim (log injection, CWE-117).

What to remember: stream so memory stays flat; cap the line length to reject oversized input; strip CR/LF before comparing or logging; and treat log content as untrusted. These four habits turn a toy counter into a parser that survives real, hostile data.

Practice with these exercises

• Count lines at a level
• First error line number
• Final Project: log-aggregator — count by level
• Detect repeated failed logins
• Redact email addresses in a log line
• Redact credit-card-like number runs
• Count suspicious IPs in a sample log