Parsing CSV

Overview

Introduction

CSV stands for Comma-Separated Values. It is one of the oldest and most common ways to move tabular data between programs: spreadsheets, databases, log exporters, bank statements, and scientific instruments all speak CSV. Each line of text is one record (think: one row), and within a line the values are separated by commas into fields (think: columns).

A tiny CSV file looks like this:

name,age,city
Alice,30,Paris
Bob,25,Berlin

The appeal is that it is just plain text. You can open it in any editor, and reading it in C is mostly an exercise in the skills you already have. This lesson builds directly on fgets for safe line reading (you read the file one bounded line at a time) and on C strings (each field is a NUL-terminated char array, and you manipulate it with pointers and the <string.h> functions).

Where the difficulty hides

CSV looks trivial: read a line, split on commas, done. For perfectly clean data that is true. But real CSV files are produced by spreadsheets and other tools that follow a convention (loosely described by RFC 4180) where a field may be wrapped in double quotes so it can itself contain a comma, a quote, or even a newline:

id,full_name,note
1,"Smith, John","He said ""hi"""

Here line 2 has exactly three fields, even though it contains three commas and several quote characters. A naive split on , would wrongly report five fields. Most of this lesson is about understanding that gap between the simple model and real data, and writing code that survives it.

Terminology

• Record / row: one line of the file (subject to the embedded-newline exception below).
• Field / column: one value within a record.
• Delimiter: the character separating fields — a comma here, but TSV files use a tab, and European exports sometimes use ;.
• Quoting: wrapping a field in " so special characters inside it are treated as data.
• Escaped quote: a literal " inside a quoted field, written as two quotes "".

Why it matters

Why this matters

CSV is the universal lowest-common-denominator data format. If you write software that imports user-uploaded data, exports reports, ingests bank or payment records, or feeds a data pipeline, you will parse CSV. Getting it wrong is not a cosmetic bug — it shifts every column after the mistake, so an address ends up in the phone-number field and a price lands in the quantity column. That kind of error is silent: the program does not crash, it just stores wrong data.

In C specifically, CSV parsing is a focused way to practice the disciplines that prevent the most common security and stability bugs: reading input with a fixed bound, never assuming a field fits a buffer, and treating every byte of external input as untrusted. A loose strcpy of a CSV field into a fixed array is a textbook buffer overflow. So even though CSV is not a "security topic", parsing it well is exactly the muscle you use to write safe input handling everywhere else.

Core concepts

1. The naive model and why it is tempting

Definition. The naive model treats a CSV line as "text split on commas": every comma is a field boundary, and the pieces between commas are the fields.

For clean data it is correct and one line of code:

char *tok = strtok(line, ",");
while (tok) { /* use tok */ tok = strtok(NULL, ","); }

Why it is tempting: it is short, and it works on the data you typed by hand to test it. Why it fails: real exporters quote fields, and a quoted field can contain commas, quotes, and newlines. The naive model has no concept of "inside a quote", so it cuts those fields apart.

Knowledge check: Given the line 1,"Smith, John",30, how many fields does a human see, and how many would a plain comma-split report? Why do they differ?

2. strtok and strtok_r: the destructive tokenizer

Definition. strtok walks a string and returns one token at a time, treating any of the delimiter characters as a separator. It works by overwriting each delimiter with a \0 so the returned pointer looks like a normal C string, and by remembering where it stopped so the next call can continue.

How it works internally — memory view. Starting with "Alice,30,Paris":

before:  A l i c e , 3 0 , P a r i s \0
after:   A l i c e \0 3 0 \0 P a r i s \0
                   ^         ^
              comma -> \0  comma -> \0
returns: "Alice"  then  "30"  then  "Paris"

Two consequences you must remember:

• It mutates the buffer you pass in. You can never tokenize a string literal (char *s = "a,b"; then strtok(s, ...) is undefined behaviour — the literal is read-only). Use a writable array or a copy.
• Plain strtok keeps that "where I stopped" position in a single hidden static variable, so it is not re-entrant and not thread-safe, and you cannot interleave two tokenizations. The fix is strtok_r (POSIX) or strtok_s (C11 Annex K / Windows), which keep the position in a char *saveptr you own:

char *save = NULL;
char *tok = strtok_r(line, ",", &save);
while (tok) { tok = strtok_r(NULL, ",", &save); }

A subtle pitfall: strtok treats runs of delimiters as one separator and skips leading/trailing delimiters, so it silently drops empty fields. For "a,,b" it returns only "a" and "b" — the empty middle field vanishes. For CSV, where empty fields are meaningful, this is a real bug.

When to use it: quick parsing of clean, unquoted, no-empty-field data, or as a learning tool. When NOT to: any data that may be quoted, may contain empty fields, or is shared across threads.

Knowledge check (predict the output): You run strtok_r with delimiter "," over the buffer "x,,y". How many tokens come back, and what are they? Is that what a CSV reader should produce?

3. Quoting and escaped quotes

Definition. A CSV field may be enclosed in double quotes. Inside the quotes, a comma and a newline are ordinary data. A literal double-quote inside a quoted field is written as two double-quotes (""), which represent one " in the value.

Structure of one quoted field:

"He said ""hi"" to me"
 ^        ^^   ^^      ^
 open    esc  esc    close
value ->  He said "hi" to me

Rules a correct reader follows: a " right after a field boundary starts a quoted field; inside it everything is literal until the next "; if that " is immediately followed by another ", it is an escaped quote (emit one ", stay inside); otherwise it closes the field, and the next character should be a delimiter or end-of-line.

When you need this: any file produced by Excel, Google Sheets, a database export, or another standards-following tool. When you can skip it: strictly internal data you generated yourself and guarantee never contains commas, quotes, or newlines in a field.

4. Embedded newlines and the "one line = one record" trap

Because a quoted field may contain a newline, a single record can span several physical lines. That means "read one line, parse it" is only safe for unquoted data. A fully correct reader counts quotes: if a line ends while still inside an open quote, it must read and append the next line before parsing the record.

physical lines:        logical record:
1,"line one        ->  field0 = 1
still same field"      field1 = "line one\nstill same field"

5. Line endings: \n vs \r\n

fgets keeps the trailing newline. On Unix that is \n; on files created on Windows it is \r\n (carriage return + line feed). If you strip only \n, a stray \r clings to the end of the last field and quietly corrupts comparisons ("Paris\r" != "Paris"). Always strip both:

line[strcspn(line, "\r\n")] = '\0';  /* truncate at first CR or LF */

Knowledge check (find-the-bug): A program compares the last CSV field with strcmp(field, "yes") and it never matches on a file emailed from a Windows user, even though the field clearly reads yes. What is almost certainly stuck on the end of field, and which one line fixes it?

Syntax notes

Syntax you will use

#include <string.h>

/* Re-entrant tokenizer: you own the save pointer. */
char *strtok_r(char *str, const char *delim, char **saveptr);
/*  - First call: pass the buffer in `str`.
 *  - Later calls: pass NULL in `str`, same `saveptr`.
 *  - Returns the next token, or NULL when done.
 *  - DESTRUCTIVE: writes '\0' over each delimiter in `str`. */

/* Length of the prefix of s that contains NONE of the chars in reject.
   Handy for trimming line endings. */
size_t strcspn(const char *s, const char *reject);
line[strcspn(line, "\r\n")] = '\0';

/* Bounded copy of a field into a fixed buffer. snprintf NUL-terminates
   and never writes past the size, unlike strcpy. */
int snprintf(char *dst, size_t size, const char *fmt, ...);
snprintf(out, outsz, "%s", field);

Key rules: the buffer you tokenize must be writable (an array or a copy, never a string literal); you must keep one saveptr per independent tokenization; and you must check every return value (fopen, fgets, allocations) before using it.

Lesson

CSV looks simple — until it isn't

CSV (Comma-Separated Values) seems easy: split each line on commas, and you have your fields.

A quick approach like strtok(line, ",") does work for plain input. But it breaks the moment the data gets realistic.

It fails on:

• Quoted fields — a value wrapped in quotes, such as "Smith, John", where the comma is part of the data, not a separator.
• Escaped quotes — a quote character that appears inside a quoted field.
• Embedded newlines — a single field that spans more than one line.
• Trailing empty fields — a line that ends with a comma and an empty final value.

What to use

• Production code: use a tested library, such as libcsv. It already handles the tricky cases for you.
• Controlled input: if you know the data format and want the practice, write a hand parser. One that handles "…,…" quoting is about 50 lines of C — and a great exercise.

Code examples

#include <stdio.h>
#include <string.h>
#include <stdbool.h>

#define MAX_LINE   1024
#define MAX_FIELDS 64

/*
 * Parse ONE CSV record that is already a NUL-terminated line (line endings
 * stripped). Handles double-quoted fields, escaped quotes (""), and commas
 * inside quotes. Embedded newlines across physical lines are out of scope
 * here (see the lesson notes); this parses a single logical line.
 *
 * Writes pointers to each field's start into fields[]; the parse is done
 * IN PLACE by shifting characters and inserting '\0', so `line` is mutated.
 * Returns the number of fields, or -1 if there are too many for `fields`.
 */
static int parse_csv_line(char *line, char *fields[], int max_fields)
{
    int count = 0;
    char *read = line;   /* where we read from */
    char *write = line;  /* where we write the cleaned field */

    while (1) {
        if (count >= max_fields)
            return -1;               /* refuse to overflow fields[] */

        fields[count++] = write;     /* this field starts here */
        bool in_quotes = false;
        bool more_fields = false;    /* did we stop on a comma? */

        while (*read) {
            char c = *read;

            if (in_quotes) {
                if (c == '"') {
                    if (read[1] == '"') {   /* "" -> one literal quote */
                        *write++ = '"';
                        read += 2;
                        continue;
                    }
                    in_quotes = false;       /* closing quote */
                    read++;
                    continue;
                }
                *write++ = c;                /* literal char inside quotes */
                read++;
            } else {
                if (c == '"') {              /* opening quote */
                    in_quotes = true;
                    read++;
                } else if (c == ',') {       /* field boundary */
                    read++;                  /* consume the comma NOW, before */
                    more_fields = true;      /* the '\0' below overwrites it  */
                    break;
                } else {
                    *write++ = c;
                    read++;
                }
            }
        }

        *write++ = '\0';                      /* terminate this field */

        if (more_fields)                     /* another field follows */
            continue;
        break;                               /* end of line */
    }
    return count;
}

int main(void)
{
    char line[MAX_LINE];
    char *fields[MAX_FIELDS];

    /* Read CSV records from stdin so the example is self-contained. */
    while (fgets(line, sizeof line, stdin)) {
        line[strcspn(line, "\r\n")] = '\0';   /* strip \n and \r\n */

        int n = parse_csv_line(line, fields, MAX_FIELDS);
        if (n < 0) {
            fprintf(stderr, "too many fields, skipping line\n");
            continue;
        }

        for (int i = 0; i < n; i++)
            printf("[%d] '%s'\n", i, fields[i]);
        printf("--- %d field(s) ---\n", n);
    }
    return 0;
}

What it does. It reads CSV lines from standard input, strips the line ending, and parses each line with a small state machine that tracks whether it is currently inside double quotes. It compacts each field in place (turning "" into ", dropping the surrounding quotes) and prints every field with its index.

Expected output. Given the input line 1,"Smith, John","He said ""hi""" the program prints:

[0] '1'
[1] 'Smith, John'
[2] 'He said "hi"'
--- 3 field(s) ---

Edge cases handled: commas inside quotes stay in the field; "" becomes a single "; a trailing comma yields a final empty field (a,b, -> three fields, last one empty); too many fields are rejected rather than overflowing fields[]. Out of scope on purpose: a quoted field that contains a real newline spanning two physical lines (you would need to read and append the next line first), and malformed input such as an unterminated quote (this parser just consumes to end of line).

Line by line

Walkthrough of the example

We trace parse_csv_line on the buffer 1,"Smith, John" (a two-field line) to see how the in-place compaction works. read scans forward; write lays down the cleaned bytes; they share the same buffer and write never gets ahead of read.

Now fields[0] points at 1, fields[1] points at Smith, John, and the function returns 2. Notice the comma inside the quotes survived as data, and the surrounding quotes were dropped.

Why the in-place trick is safe: every cleaned field is the same length or shorter than the raw text (we only ever remove quote characters, never add), so write always trails read. That guarantees we never clobber a byte we still need to read.

The main loop ties it together: fgets reads at most sizeof line - 1 bytes (no overflow), strcspn(line, "\r\n") returns the index of the first carriage return or line feed (or end of string), and assigning '\0' there truncates the newline. Then each parsed field is printed by index.

Common mistakes

Common mistakes

1. Tokenizing a string literal

/* WRONG: literals are read-only; strtok tries to write into it */
char *line = "Alice,30,Paris";
char *tok = strtok(line, ",");   /* undefined behaviour, often a crash */

Why wrong: strtok overwrites delimiters with \0, but a string literal lives in read-only memory. Fix: use a writable array (or a copy):

char line[] = "Alice,30,Paris";   /* writable array */
char *save = NULL;
char *tok = strtok_r(line, ",", &save);

Recognize it: a segfault on the first token, only with literal input.

2. Expecting empty fields from strtok

/* WRONG for CSV: "a,,b" loses the empty middle field */
for (char *t = strtok(line, ","); t; t = strtok(NULL, ",")) ...

Why wrong: strtok collapses consecutive delimiters and skips empty fields, so a column count is off and later columns shift. Fix: use a parser that treats each comma as a boundary (like the example above), which preserves empty fields.

3. Copying a field with strcpy into a fixed buffer

char name[16];
strcpy(name, field);   /* WRONG: a long field overflows `name` */

Why wrong: a CSV field can be any length; strcpy writes until the source \0, smashing the stack. Fix: bound it.

snprintf(name, sizeof name, "%s", field);  /* never writes past `name` */

Recognize it: mysterious corruption or crashes that depend on input length — a classic buffer overflow.

4. Stripping only \n

line[strlen(line) - 1] = '\0';   /* WRONG: leaves \r on Windows files,
                                    and crashes on an empty line */

Fix: line[strcspn(line, "\r\n")] = '\0'; handles \n, \r\n, and lines with no newline at all.

Debugging tips

Debugging this topic

Compiler errors / warnings

• implicit declaration of strtok_r: it is POSIX, not plain ISO C. Compile with -D_POSIX_C_SOURCE=200809L (or -std=gnu11), or use the portable hand parser shown here.
• assignment makes pointer from integer: you likely forgot to #include <string.h>, so the compiler assumes strtok/strcspn return int.

Runtime errors

• Segfault on the first token: you tokenized a string literal — use a writable buffer (Mistake 1).
• Segfault inside the loop: you indexed fields[i] for an i the parser never filled, or you read past the count it returned. Always loop 0 .. n-1.

Logic errors

• Wrong field count: print every field with its index (the example already does [%d] '%s'). If a field shows a comma you expected to be a separator, your quote handling is off; if an empty field disappeared, you are using strtok.
• A trailing \r ghost: if strcmp never matches a value that looks correct, print the field with visible bounds like printf("'%s'\n", f) — a stray 'Paris\r' will show the issue, or dump bytes with od -c.

Questions to ask when it does not work

1. Is my input buffer writable, and large enough (fgets bound vs longest line)?
2. Did I strip both \r and \n?
3. Am I inside a quote when I think I am? Print the in_quotes state at each comma.
4. Does my field count match the number of commas plus one for unquoted lines?

Memory safety

Memory safety and undefined behaviour

CSV parsing is mostly an exercise in safe input handling. The risks specific to this topic:

• Bounded reads. Use fgets(line, sizeof line, fp) — never gets (removed from C11) and never an unbounded scanf("%s"). Decide what happens if a line is longer than your buffer: fgets will split it, so you may get a partial record. Detect it (no \n found) if correctness matters.
• Bounded field copies. Treat every field as attacker-controlled length. Copy with snprintf(dst, sizeof dst, "%s", field) or memcpy with an explicit, checked length — never strcpy/strcat into a fixed array.
• Writing to read-only memory. The destructive tokenizer (strtok) and the in-place parser both write into the buffer. Passing a string literal is undefined behaviour; pass a writable array.
• Index and count discipline. After parsing, only touch fields[0 .. n-1]. Reading an unfilled slot is uninitialized-pointer use. The example refuses (return -1) rather than overflowing fields[] — bound your arrays the same way.
• Integer/loop bounds. If you compute offsets into the buffer, keep them within [0, length]. An off-by-one when inserting \0 can drop the last field or write one byte past the array.
• NUL-termination. Every field you hand to a string function must end in \0. The parser guarantees this by writing \0 at each boundary; if you build fields manually, do the same.

A quick way to catch these while learning: compile with -fsanitize=address,undefined -Wall -Wextra and run on messy sample files. The sanitizers report overflows and read-only writes immediately.

Real-world uses

Real-world uses

• Data import/export: spreadsheets (Excel, Google Sheets), database bulk loaders (COPY ... FROM, LOAD DATA INFILE), and BI tools all exchange CSV. A robust reader is the front door of an import pipeline.
• Finance and operations: bank and card statements, payroll, and inventory feeds are routinely CSV. A shifted column here is a real-money bug.
• Logging and analytics: many tools emit comma- or tab-separated records that batch jobs parse (closely related to the next lesson, log parsing).
• Embedded and instruments: sensors and dataloggers often dump CSV to an SD card for later analysis.

Professional best practices

Beginner habits:

• Always read with a fixed bound (fgets) and copy fields with a bounded function (snprintf).
• Strip both \r and \n.
• Validate the field count per row before trusting columns; reject or log malformed rows instead of guessing.
• Check every return value: fopen, fgets, allocations.

Advanced habits:

• For production, prefer a tested library such as libcsv; it already handles quoting, escapes, embedded newlines, and CRLF, and has been fuzzed. Hand-roll only for controlled input or for learning.
• Decide the dialect explicitly: delimiter (, vs ; vs tab), quote char, and whether the first row is a header. Do not hard-code assumptions silently.
• Stream large files line by line; do not load a multi-gigabyte CSV into memory.
• Treat all input as untrusted — bound everything, and never eval/execute a field value.
• Free or close everything you open (fclose the file) and run under ASan/UBSan in CI.

Practice tasks

Practice tasks

Beginner 1 — Count fields (no quotes)

Objective: count the comma-separated fields in a clean line.

Requirements: write int count_fields(const char *line) returning the number of fields, defined as the number of commas plus one (an empty line has one empty field).

Example: "a,b,c" -> 3; "a,,b" -> 3; "" -> 1.

Constraints: do not modify line; do not use strtok (it would miss empty fields). Hint: loop once, increment a counter on each ,. Concepts: string traversal, why empty fields matter.

Beginner 2 — Strip the line ending

Objective: safely remove a trailing \n or \r\n.

Requirements: write void chomp(char *s) that truncates s at the first \r or \n. It must do nothing harmful on an empty string.

Example: "Paris\r\n" -> "Paris"; "Paris" -> "Paris"; "" -> "".

Constraints: one pass, no out-of-bounds access. Hint: strcspn(s, "\r\n") gives the index to terminate at. Concepts: strcspn, NUL-termination.

Intermediate 1 — Extract field by index (unquoted)

Objective: copy field idx from a simple line into a bounded buffer.

Requirements: write int get_field(const char *line, int idx, char *out, int outsz) that copies the 0-based field into out (always NUL-terminated, never exceeding outsz), returns 0 on success, -1 if idx is out of range.

Example: get_field("a,bb,ccc", 1, out, sizeof out) -> out == "bb", returns 0.

Constraints: never write past outsz; do not modify line. Hint: walk commas to find the start/end of field idx, then copy with a length check (or snprintf with "%.*s"). Concepts: bounded copy, off-by-one care.

Intermediate 2 — Quote-aware field count

Objective: count fields correctly when fields may be quoted.

Requirements: extend Beginner 1 so commas inside double quotes do not count as separators; "" inside quotes is an escaped quote, not a closer.

Example: 1,"Smith, John",30 -> 3; "a""b",c -> 2.

Constraints: single pass, track an in_quotes flag. Hint: reuse the state-machine logic from the lesson example but only count, do not copy. Concepts: state machine, quoting/escaping.

Challenge — Multi-line record reader

Objective: read a full logical CSV record that may span several physical lines because a quoted field contains a newline.

Requirements: write a function that, given a FILE *, returns the next complete record as one string (or NULL at EOF). Keep reading and appending lines while the number of unescaped quotes seen so far is odd (meaning a quote is still open). Then parse it with your quote-aware parser.

Example input:

1,"line one
still same field",ok

Expected: one record with three fields, field 1 = line one\nstill same field.

Constraints: bounded buffers throughout; handle EOF mid-quote gracefully (treat as end of record). Hint: count quotes per line to decide whether to keep reading; remember "" is two quotes (even count, stays balanced). Concepts: state across lines, dynamic or bounded growth, RFC-4180-style quoting.

Summary

Summary

• CSV is plain text organized as records (lines) and fields (comma-separated values). It builds on safe line reading (fgets) and C string handling.
• The naive strtok(line, ",") works only for clean data. It is destructive (writes \0 over delimiters, so never on a string literal), not re-entrant (prefer strtok_r), and it silently drops empty fields — wrong for CSV.
• Real CSV uses quoting: a field wrapped in " may contain commas and newlines; a literal quote is written "". Correct parsing needs a small state machine that tracks whether it is inside quotes.
• Most important syntax: strtok_r(str, delim, &saveptr), line[strcspn(line, "\r\n")] = '\0' to strip endings, and snprintf(dst, sizeof dst, "%s", field) for bounded field copies.
• Common mistakes: tokenizing a literal, expecting empty fields from strtok, strcpy into a fixed buffer (overflow), and stripping only \n (leaving \r).
• Remember: treat every field as untrusted, bound every read and copy, count quotes to handle embedded newlines, and for production reach for a tested library like libcsv rather than hand-rolling.

Practice with these exercises

• Count CSV fields
• Extract a CSV field