File descriptors

Overview

In the prerequisite lesson fopen and the stdio model you opened files with fopen, which gives you a FILE * and a friendly, buffered API (fprintf, fgets, fread). That convenience layer is built on something thinner and more general underneath: the file descriptor.

A file descriptor (almost always shortened to fd) is a small non-negative integer. The kernel uses it as a name for something your program has open. That "something" does not have to be a file on disk — it can be a regular file, a pipe, a network socket, a terminal, or a device. The genius of Unix is that all of these are reached through the same integer handle and the same handful of calls. "Everything is a file" really means "almost everything is reachable through a file descriptor."

Every process starts life with three descriptors already open and connected to its terminal (or to whatever the shell redirected them to):

• 0 — standard input (stdin)
• 1 — standard output (stdout)
• 2 — standard error (stderr)

That is why printf (which writes to fd 1) and perror (which writes to fd 2) work without you opening anything. You create more descriptors with open, you do I/O on them with read and write, and you give them back to the kernel with close.

Terminology you will meet here: the descriptor is the integer in your process; the kernel keeps a per-process file-descriptor table mapping each integer to an internal open file description (the thing that tracks the current offset and access mode); and that, in turn, points to the underlying file or device. Keeping those three layers straight is the key to understanding inheritance and redirection later in this lesson.

Why it matters

Beneath the familiar fopen API lives a thinner, more powerful layer: raw file descriptors. This is how the kernel exposes every I/O object — files, pipes, sockets, terminals — through one uniform integer handle.

If you ever want to:

• write a shell that supports <, >, and | (these are all descriptor tricks),
• write a network server that juggles hundreds of client connections,
• do non-blocking or event-driven I/O with select/poll/epoll,
• or simply understand error messages like "too many open files" in production,

...you need to work at the file-descriptor layer. Stdio hides descriptors from you on purpose, but the moment you step outside plain file reading and writing, the abstraction leaks and you must understand what is underneath. Descriptors are also where many real-world resource bugs live: a server that forgets to close a client socket will slowly exhaust its descriptor limit and stop accepting connections — a failure that looks mysterious until you know to look at fds.

Core concepts

1. The descriptor is an index, not a pointer

Definition. A file descriptor is a small non-negative integer that the kernel uses as an index into your process's open-file table.

How it works internally. Each process has its own array of "open file" slots. Slot 0 is stdin, slot 1 is stdout, slot 2 is stderr, and the rest start empty. When you call open, the kernel fills the lowest-numbered free slot and returns that index to you. Because the number is just a table index, it is meaningless in another process — fd 5 in your program and fd 5 in mine point to completely different things.

Process A                         Kernel
+-----------------------+         +------------------------------+
| fd table              |         | open file descriptions       |
|  0 ->  ----------------+-------->| stdin  (terminal)            |
|  1 ->  ----------------+-------->| stdout (terminal)            |
|  2 ->  ----------------+-------->| stderr (terminal)            |
|  3 ->  ----------------+-------->| data.bin, offset=0, O_RDONLY |
|  4 ->  (free)          |         +------------------------------+
+-----------------------+

When to use it. Whenever you need the kernel's raw I/O object: sockets, pipes, polling many sources, non-blocking I/O, or precise control over flags. When not to. For ordinary line-by-line or formatted file work, stdio (fopen/fprintf/fgets) is friendlier and buffers for you.

Pitfall. Treating the descriptor number as stable or meaningful. After a close, that number is free again and the next open may reuse it — so a stale copy of an old fd can silently point at a brand-new file.

Knowledge check. A process has stdin/stdout/stderr open, then calls open twice successfully. What two numbers does it most likely get back, and why those?

2. The lowest-free-descriptor rule

Definition. open (and dup, pipe, socket, etc.) always returns the smallest descriptor number not currently in use.

Why it matters. This deterministic rule is what makes shell redirection possible. If you close(1) and then open("out.txt", ...), the new file lands in slot 1 — so anything that writes to stdout now writes to the file, with no change to the writing code.

Before:                 close(1):               open("out.txt"):
 0 stdin                0 stdin                 0 stdin
 1 stdout               1 (free)  <--lowest     1 out.txt   <-- reused!
 2 stderr               2 stderr                2 stderr

Pitfall. Relying on a specific number without checking. Always use the return value of open; never assume "it will be fd 3."

Knowledge check (predict the output). If fd1 = open(...) returns 3, you then close(3), then fd2 = open(...) again — what number is fd2 likely to be?

3. Inheritance across fork

Definition. When a process calls fork to create a child, the child receives a copy of the parent's descriptor table. The numbers are the same and they point at the same open file descriptions.

How it works. The table entries are copied, but the underlying open file description (including the shared read/write offset) is shared. So if parent and child both write to an inherited fd, their output interleaves into the same file and they advance the same offset. This sharing is exactly how a shell wires a pipe between two commands.

Pitfall. Forgetting to close the ends you don't need after fork. In a pipe, the child must close the write end (and the parent the read end) or the reader never sees end-of-file. You will build on this in the pipe-chain exercise.

4. Redirection with dup and dup2

Definition. dup2(oldfd, newfd) makes newfd refer to the same open file as oldfd, closing whatever newfd was first. dup(oldfd) is the same idea but lets the kernel pick the lowest free number.

How it works. After dup2(file_fd, 1), descriptor 1 (stdout) and file_fd both point at the same target. A later printf (which writes to fd 1) now lands in the file. This is cleaner than the close-then-open trick because it is atomic and you choose the target number.

Before dup2(file_fd, 1):        After dup2(file_fd, 1):
  1 -> terminal                  1 -> out.txt   (old terminal entry closed)
  4 -> out.txt                   4 -> out.txt

When to use. Output/input redirection (>, <), connecting a child to a pipe, logging stderr to a file. When not to. When you just want a second handle in the same process — usually you can pass the fd around instead.

Pitfall. Calling dup2(fd, 1) and then forgetting that the original fd is still open. After redirecting, you typically close(fd) so only 1 keeps the target.

Knowledge check (explain in your own words). Why does dup2(file_fd, 1) let an unchanged printf write into a file? Mention fd 1 in your answer.

5. Closing and the descriptor limit

Definition. close(fd) releases the slot back to the kernel. Every successful open needs a matching close.

How it works. Each process has a limit on open descriptors (often 1024 by ulimit -n default; servers raise it). A descriptor that is opened but never closed is leaked: the slot stays used. Leak enough and the next open/accept fails with EMFILE ("too many open files").

Pitfall. Early return on an error path that skips close. Make cleanup unconditional (a single cleanup block, or if (fd >= 0) close(fd);).

Syntax notes

The four core calls and their signatures:

#include <fcntl.h>   /* open, O_RDONLY, O_WRONLY, O_CREAT, O_TRUNC ... */
#include <unistd.h>  /* read, write, close, dup2 */

int fd = open("file.txt", O_RDONLY);   /* returns fd >= 0, or -1 on error  */
if (fd < 0) { perror("open"); return 1; }

char buf[4096];
ssize_t n = read(fd, buf, sizeof buf);  /* returns bytes read, 0 = EOF, -1 = error */
if (n < 0) { perror("read"); close(fd); return 1; }

ssize_t w = write(1, buf, (size_t)n);   /* fd 1 = stdout; returns bytes written or -1 */
if (w < 0) { perror("write"); close(fd); return 1; }

close(fd);                              /* always release the descriptor    */

Key points in the signatures:

• open takes flags (O_RDONLY, O_WRONLY, O_RDWR, optionally OR-ed with O_CREAT | O_TRUNC | O_APPEND). When you pass O_CREAT you also pass a permission mode as a third argument, e.g. open(path, O_WRONLY|O_CREAT|O_TRUNC, 0644).
• read/write return a signed ssize_t. A short count is normal — read may return fewer bytes than you asked for, and write may write fewer than you gave it. Loop if you must move all the data.
• A read return of 0 means end of file, not an error.

Lesson

A file descriptor is a small non-negative integer that identifies an open file, socket, or pipe. By convention, 0 is stdin, 1 is stdout, and 2 is stderr.

The POSIX calls open, read, write, and close operate on file descriptors directly. This is the layer that <stdio.h> is built on top of.

When to use file descriptors

Drop down to file descriptors when you need precise control, such as:

• non-blocking I/O
• polling many descriptors at once
• working with sockets

For everyday file work, stdio is friendlier.

Code examples

#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

/*
 * Copy a file at the raw file-descriptor layer (no stdio buffering).
 * Usage: ./fdcopy <source> <destination>
 * Demonstrates open / read / write / close and correct short-read/short-write
 * handling, plus unconditional cleanup on every error path.
 */
int main(int argc, char **argv) {
    if (argc != 3) {
        fprintf(stderr, "usage: %s <source> <destination>\n", argv[0]);
        return 2;
    }

    int src = open(argv[1], O_RDONLY);
    if (src < 0) { perror("open source"); return 1; }

    /* Create the destination (or truncate it) with owner rw, group/other r. */
    int dst = open(argv[2], O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (dst < 0) {
        perror("open destination");
        close(src);            /* clean up the descriptor we already hold */
        return 1;
    }

    char buf[4096];
    ssize_t n;
    int status = 0;

    /* read returns 0 at EOF, -1 on error, otherwise a (possibly short) count. */
    while ((n = read(src, buf, sizeof buf)) > 0) {
        ssize_t off = 0;
        while (off < n) {                       /* write may be short: loop  */
            ssize_t w = write(dst, buf + off, (size_t)(n - off));
            if (w < 0) {
                if (errno == EINTR) continue;   /* interrupted: retry        */
                perror("write");
                status = 1;
                goto done;
            }
            off += w;
        }
    }
    if (n < 0) { perror("read"); status = 1; }

done:
    close(src);
    if (close(dst) < 0) { perror("close destination"); status = 1; }
    return status;
}

What it does. This is a minimal cp. It opens the source read-only, creates the destination, and copies the bytes in 4 KB chunks using only read/write, then closes both descriptors. Run it as ./fdcopy notes.txt copy.txt; on success it prints nothing and exits 0, and copy.txt is byte-for-byte identical to notes.txt.

Expected output and edge cases. Normal success produces no output (Unix tools are quiet on success). If the source does not exist you get open source: No such file or directory and exit code 1. Notice the inner write loop: a single write may move fewer bytes than requested, so we keep going until the whole chunk is out. We also retry on EINTR (a signal interrupting the call), close dst even on the error path via the goto done cleanup, and check close(dst) itself because buffered data can surface an error only at close time.

Line by line

Walking through the copy program:

1. Argument check. If we did not get exactly a source and a destination, print usage to fd 2 (stderr) and exit 2. Usage messages go to stderr, never stdout.
2. open(argv[1], O_RDONLY) asks the kernel for the source file, read-only. The kernel fills the lowest free slot — with 0,1,2 already taken, src is typically 3. On failure it returns -1 and sets errno, which perror turns into a readable message.
3. open(argv[2], O_WRONLY | O_CREAT | O_TRUNC, 0644) opens the destination for writing, creating it if absent and truncating it to empty if present. dst is typically 4. If this fails we must close(src) before returning, or we leak descriptor 3.
4. The outer while calls read(src, buf, 4096). Each call returns how many bytes it actually read. The table below traces a 10000-byte source:

Call        read() returns   total copied   meaning
----------  --------------   ------------   ---------------------------
1st read    4096             4096           full buffer
2nd read    4096             8192           full buffer
3rd read    1808             10000          short read near end of file
4th read    0                10000          EOF -> loop ends

5. The inner while (off < n) writes the chunk. Because write can be short, we advance off by however many bytes it actually wrote and loop until the whole chunk is flushed. EINTR means a signal interrupted us before any bytes moved, so we simply retry.
6. After the loop, n < 0 would mean a read error; we record status = 1.
7. done: is the single cleanup point. We close(src) and close(dst). Checking close(dst)'s return matters because a delayed write error can only show up here.
8. Return status — 0 for success, 1 if anything went wrong — so the shell and scripts can react.

Common mistakes

1. Forgetting to close descriptors (the leak)

/* WRONG: leaks one descriptor per call */
int read_header(const char *path) {
    int fd = open(path, O_RDONLY);
    char buf[64];
    read(fd, buf, sizeof buf);
    return buf[0];           /* never closed! */
}

Call this in a loop and you eventually hit EMFILE. Why it is wrong: the slot is never returned, so the table fills up. Fix: close before every return.

int read_header(const char *path) {
    int fd = open(path, O_RDONLY);
    if (fd < 0) return -1;
    char buf[64];
    ssize_t n = read(fd, buf, sizeof buf);
    close(fd);               /* always */
    return n > 0 ? buf[0] : -1;
}

Recognize it: rising fd count in lsof -p <pid> or ls /proc/<pid>/fd.

2. Not checking the return value of open/read/write

int fd = open(path, O_RDONLY);   /* could be -1 */
read(fd, buf, n);                /* read(-1, ...) fails silently */

Why it is wrong: open returns -1 on failure; passing -1 to read just fails too, and you never learn the real reason. Fix: check if (fd < 0) { perror("open"); ... } right away.

3. Assuming read fills the buffer

char buf[100];
read(fd, buf, 100);
buf[100] = '\0';   /* off-by-one AND assumes 100 bytes were read */

Why it is wrong: read may return fewer than 100 bytes, and writing buf[100] is out of bounds. Fix: use the returned count and a buffer with room: ssize_t n = read(fd, buf, sizeof buf - 1); if (n >= 0) buf[n] = '\0';.

4. Mixing a FILE * and a raw fd for the same file

Writing through fprintf(fp, ...) (buffered) and write(fileno(fp), ...) (unbuffered) on the same file interleaves output in the wrong order. Fix: call fflush(fp) before dropping to the raw fd, or pick one layer and stay there.

Debugging tips

Compiler errors

• "implicit declaration of function 'open'/'read'/'close'" — you forgot #include <fcntl.h> (for open and the O_* flags) and/or #include <unistd.h> (for read/write/close).
• "'O_CREAT' undeclared" — same fix: include <fcntl.h>.

Runtime errors (check errno / perror)

• ENOENT — the path does not exist (and you did not pass O_CREAT).
• EACCES — permission denied for that mode.
• EMFILE / ENFILE — too many open files: you are leaking descriptors. Audit every open for a matching close.
• EBADF — you used a descriptor that was never opened or was already closed (often a stale fd after close).

Logic errors

• Output appears truncated → you ignored a short write; loop until all bytes are sent.
• File looks empty after writing → you forgot the program exited before flushing, or you opened with O_TRUNC and never wrote.

Tools that make descriptors visible

• lsof -p <pid> — lists every open descriptor of a running process; the go-to for hunting leaks.
• strace -e trace=openat,read,write,close ./prog — prints each I/O system call as it happens, with arguments and return values.
• /proc/<pid>/fd/ (Linux) — a directory whose entries are the process's live descriptors, each a symlink to its target.

Questions to ask when it doesn't work: Did open return -1? Did I check it? Is the fd I'm using the return value of open, or a guessed number? Did I close on every path? Am I mixing stdio and raw I/O?

Memory safety

This is not a security lesson, so the focus here is robustness and resource safety:

• Resource leaks, not memory leaks. A descriptor leak is a resource leak — Valgrind will not catch it. The symptom is EMFILE after the program has run for a while. The rule stays simple: every successful open needs a close, on every path. For risky code, guard with if (fd >= 0) close(fd);.
• Buffer bounds with read. read(fd, buf, sizeof buf) is safe only when sizeof buf is the real capacity. Passing a larger length than the buffer holds is a buffer overflow (undefined behavior). Never null-terminate at a fixed index — terminate at the returned count, and leave one byte of room (sizeof buf - 1).
• Uninitialized data. If read returns a short count, only the first n bytes of the buffer are valid. Reading past n reads uninitialized stack memory.
• Use the return values. read/write return ssize_t; ignoring a -1 or a short count leads to silent data loss or, worse, decisions based on garbage. Treat 0 from read as EOF, not error.
• Stale descriptors. After close(fd), do not reuse fd; the number can be handed to the next open, so an old copy may now point at an unrelated file (EBADF if lucky, silent corruption if not).

Real-world uses

Where descriptors show up in real systems

• Shells implement <, >, >>, and | entirely with open, dup2, close, and pipe — exactly the calls in this lesson.
• Web and database servers keep one descriptor per client connection (each socket is an fd) and multiplex thousands at once with epoll/kqueue.
• The tee command writes one input stream to several output descriptors at once (mirrored by the tee-write-both exercise).
• Containers (Docker, etc.) and sandboxes pass descriptors between processes and keep resources reachable by fd even after the path changes.
• Logging frameworks redirect stderr to a log file with dup2 so existing perror/fprintf(stderr, ...) calls keep working unchanged.

Professional best practices

Beginner habits:

• Always check the return of open, read, write, and close.
• Pair every open with a close; prefer a single cleanup point.
• Use the returned descriptor and returned byte count — never guess.
• Send errors and usage to stderr (fd 2), data to stdout (fd 1).

Advanced habits:

• Loop on short reads/writes when you must move an exact amount.
• Handle EINTR (retry) on slow calls.
• Set O_CLOEXEC on descriptors that should not survive an exec to avoid leaking them into child programs.
• Raise the descriptor limit deliberately for high-connection servers and monitor fd usage as a health metric.

Practice tasks

Beginner 1 — Read the first 100 bytes. Write a program that takes a filename, opens it O_RDONLY, reads up to 100 bytes into a buffer, writes them to stdout (fd 1), and closes the file. Requirements: check open and read; null-terminate safely if you print as a string. Hint: read may return fewer than 100 bytes — use the count. Concepts: open, read, write, close.

Beginner 2 — Count open descriptors in a table. Given an array of int where each slot is a live fd or -1 for closed, return how many are live. Input: {3, -1, 5, -1, 7} → Output: 3. Hint: a single loop counting entries >= 0. Concepts: the descriptor table as an array; the lowest-free rule's mental model. (Mirrors the Count open descriptors exercise.)

Intermediate 1 — Redirect stdout to a file with dup2. Open out.txt for writing, call dup2(file_fd, 1), close the original file_fd, then call printf("hello\n"). Confirm the text lands in out.txt, not the terminal. Requirements: check every call; explain in a comment why printf now writes to the file. Concepts: dup2, fd 1, redirection.

Intermediate 2 — Robust file copy. Extend the lesson's copy program to copy from stdin (fd 0) to stdout (fd 1) when run with no arguments, behaving like cat. Requirements: handle short writes in a loop and EINTR. Input/Output: echo hi | ./mycat prints hi. Concepts: read/write loops, standard descriptors.

Challenge — Two-output tee. Write a program that reads stdin and writes every byte to both stdout and a file given on the command line, like the Unix tee. Requirements: open the file with O_WRONLY|O_CREAT|O_TRUNC, 0644; for each chunk read, fully write it to fd 1 and to the file fd, each with its own short-write loop; close the file at the end and check the result; report partial failures. Constraints: no stdio for the data path — only read/write. Hint: one read, two write loops. Concepts: multiple descriptors, short-write handling, cleanup. (Related to tee-write-both.)

Summary

• A file descriptor is the kernel's uniform, small-integer handle for every I/O object: files, pipes, sockets, terminals. It is an index into your process's open-file table, not a pointer or a name.
• Three descriptors are always open at start: 0 stdin, 1 stdout, 2 stderr. The kernel hands out the lowest free number on each open.
• The core calls are open (→ fd or -1), read (→ bytes, 0 = EOF, -1 = error), write (→ bytes or -1), and close. Always check returns; read/write may be short, so loop when you must move exact amounts.
• dup2(oldfd, newfd) makes newfd point where oldfd points — the basis of shell redirection (dup2(file_fd, 1) sends stdout into a file). fork copies the whole table, which is how pipes get wired.
• The most common mistakes: forgetting close (leaks → EMFILE), ignoring return values, assuming read fills the buffer, and mixing a FILE * with a raw fd without fflush.
• Remember: open, use, close — never leak a descriptor; descriptors are the foundation beneath fopen, pipes, sockets, and the whole Unix I/O model. Drop to descriptors for precise control; use stdio for everyday convenience.

Practice with these exercises

• Lowest free file descriptor
• Count open descriptors
• Plan a pipe chain for N processes
• Write a buffer to two outputs