Linux System Programming · advanced · ~12 min

Pipes

Connect two processes with a unidirectional byte stream.

Overview

A pipe is a one-way data channel managed by the kernel. Data written at one end can be read at the other.

It comes with two file descriptors:

  • One for writing into the pipe.
  • One for reading out of the pipe.

The call pipe(fds) fills the array for you: fds[0] becomes the read end, and fds[1] becomes the write end.

Combined with fork (creating a child process) and dup2 (redirecting a file descriptor), pipes let you build shell pipelines such as ls | grep foo.

Why it matters

Pipes are how Unix programs cooperate.

Every | you type in a shell creates a pipe between two processes. The output of one command flows straight into the input of the next.

Pipes also let a parent and child process exchange data inside a single program. For example, a parent can read whatever its child writes to standard output.

Core concepts

Atomic writes up to PIPE_BUF

Small writes (4 KB or less on Linux) are guaranteed to arrive in one piece, never interleaved with another writer's data. This is useful for log collectors that gather output from many sources.

EOF on read

When the write end is fully closed, a read on the pipe returns 0. That zero is the signal for end-of-file (EOF).

Blocking by default

  • Reading from an empty pipe blocks until data arrives or the write end closes.
  • Writing to a full pipe blocks until space frees up.

Close the ends you don't use

After a fork, the child inherits both ends of the pipe. Close every end you do not actually use.

If you leave a write end open somewhere, the reader will never see EOF, because the kernel still thinks more data could arrive.

Syntax notes

int fds[2];
pipe(fds);                  // fds[0] = read end, fds[1] = write end
if (fork() == 0) {           // child
    dup2(fds[1], 1);          // stdout -> pipe write end
    close(fds[0]); close(fds[1]);
    execlp("ls", "ls", NULL);
}
// parent:
close(fds[1]);               // don't keep the write end alive
char buf[4096];
read(fds[0], buf, sizeof buf);

Lesson

int pipe(int p[2]) creates a pair of file descriptors:

  • p[0] to read from.
  • p[1] to write to.

The usual pattern is:

  1. Call fork to create a child process.
  2. Use dup2 to attach the right pipe end onto stdin or stdout.
  3. Close the unused end.
  4. Call exec to run the target program.

A shell pipeline a | b is exactly this sequence of steps.

Code examples

int p[2]; pipe(p);
if (fork() == 0) {        // child
    dup2(p[0], 0); close(p[0]); close(p[1]);
    execvp("wc", (char*[]){"wc","-l", NULL});
    _exit(127);
}
close(p[0]);
write(p[1], data, n); close(p[1]);
wait(NULL);

Common mistakes

  • Forgetting to close the unused end of the pipe. The reader then blocks forever, waiting for an EOF that never comes because some write end is still open.

Debugging tips

Reader never sees EOF

You forgot to close the write end. Check the parent, and check both sides. Close every write end you do not actually use.

Inspect every pipe operation

strace -f -e trace=pipe,read,write,close ./your_program

This prints each pipe, read, write, and close call as it happens.

Two processes deadlock

This usually means each side is waiting to write to the other at the same time, and neither is reading. Use two separate pipes, one for each direction.

Memory safety

Pipes are file descriptors, so they leak the same way any file descriptor does. Always close both ends when you are done.

Reading into a buffer of size N reads at most N bytes. Always check the return value: a short read (fewer bytes than you asked for) is normal on pipes, not an error.

Real-world uses

  • Shell pipelines, such as grep foo big.log | wc -l.
  • Inter-process communication between cooperating programs.
  • Capturing a subprocess's standard output.
  • Building command pipelines programmatically from your own code.

Practice tasks

  1. Create a pipe and fork. Have the child write hello, and have the parent read it and print it.
  2. Reproduce ls | wc -l using two fork calls and two exec calls joined by one pipe.
  3. Deliberately skip closing the write end. Observe that the read blocks forever, and note why.

Summary

  • A pipe is a one-way kernel buffer with a read end (fds[0]) and a write end (fds[1]).
  • Pair it with fork and dup2 to chain processes, just like a shell pipeline.
  • Close every end you don't use, or the reader never sees EOF.
  • Check read return values: short reads are normal on pipes.

Practice with these exercises