send() and recv() — moving bytes

Overview

In the previous lesson you used connect() to establish a TCP connection and got back a connected file descriptor. Now that the pipe exists, this lesson is about pushing bytes through it. send() and recv() are the socket-specific siblings of write() and read(): same idea, plus a flags argument for socket-only behavior.

The API itself is tiny — you will learn it in two minutes. The real subject of this lesson is a property of TCP that trips up almost every beginner: TCP is a byte stream, not a message service. The kernel is free to hand you fewer bytes than you asked for, split your message across calls, or glue two messages together. Everything here builds on the connected socket from connect-syscall; we take that fd and learn to move data over it correctly.

Why it matters

Every networked program a defender touches — a web server, a logging agent, a reverse proxy, a database client — is built on loops around send/recv. The single most common networking bug in production C is assuming one recv() returns a whole message; under load or across the internet it frequently does not, causing truncated parses, hangs, and desyncs. From a security angle, trusting an attacker-supplied length prefix without bounds-checking it is a classic remote buffer overflow, and mishandling recv == 0 versus recv == -1 leads to busy-loops and denial of service. Getting these fundamentals right is the difference between a robust service and one an attacker can wedge with a slow or malformed stream.

Core concepts

The API: send and recv are write and read, plus flags

ssize_t send(int fd, const void *buf, size_t len, int flags);
ssize_t recv(int fd, void *buf, size_t cap, int flags);

With flags == 0 these behave exactly like write() and read() on the socket. Both return the number of bytes actually transferred, which can be less than you asked for. On a connected TCP socket you can freely mix send/recv with write/read; the flags argument is the only reason to prefer the socket versions.

TCP is a byte stream, not a message queue

This is the idea the whole lesson rests on. When you send(fd, buf, 1000, 0), TCP does not promise the peer will recv 1000 bytes in one call. It promises only that the bytes arrive in order, exactly once. The boundaries you sent with are erased.

  You send:      [ "HELLO" ][ "WORLD" ]      (two send() calls)

  Wire (TCP):     H E L L O W O R L D          (one ordered stream)

  Peer may recv:  [ "HELL" ]                    <- short read (4 bytes)
                  [ "OWORL" ]                    <- boundary crossed!
                  [ "D" ]

So a single recv() may return part of one message, or several messages stuck together. Your code must impose its own structure on the stream.

Reading return values correctly

The two classic mistakes both live in this table: treating recv == 0 as an error (it is normal end-of-stream), and assuming send/recv transfer everything in one shot (they may not).

Short writes and short reads: always loop

Because any single call can be short, correct code loops until the byte budget is exhausted:

ssize_t send_all(int fd, const void *buf, size_t n) {
    const char *p = buf;
    while (n > 0) {
        ssize_t k = send(fd, p, n, 0);
        if (k < 0) { if (errno == EINTR) continue; return -1; }
        p += k; n -= (size_t)k;
    }
    return 0;
}

The read side is the same shape, except it must also stop on recv == 0 (EOF).

Knowledge check: your peer sends the 8-byte word "DEFENDER" with one send. You call recv(fd, buf, 8, 0) and it returns 5. Is that an error, and what do you do?

Not an error. It is a short read — completely normal for TCP. You have 5 valid bytes ("DEFEN"). Call recv again with buf + 5 and remaining capacity 3; it will hand you the last 3 bytes ("DER"). Only -1 is an error and only 0 is end-of-stream.

Framing: giving structure back to the stream

Since TCP erases message boundaries, you must add them yourself. The two standard techniques are length-prefixing (send a fixed-size count, then that many payload bytes) and delimiters (e.g. HTTP ends headers with \r\n\r\n). Length-prefixing is simpler and used in the example below:

  Frame on the wire:
  +---------------------+-----------------------------+
  |  4-byte length (N)  |     N bytes of payload      |
  |  big-endian uint32  |                             |
  +---------------------+-----------------------------+

The receiver recv_alls exactly 4 bytes to learn N, then recv_alls exactly N payload bytes. Defensive rule: before trusting N, check it against your buffer capacity — an attacker who controls the length prefix will happily send 0xFFFFFFFF to make you over-read or over-allocate.

Useful flags

MSG_NOSIGNAL is the one you almost always want on send for a server: otherwise writing to a socket whose peer has vanished delivers SIGPIPE, which by default kills your process.

Syntax notes

#include <sys/socket.h>
ssize_t send(int fd, const void *buf, size_t len, int flags);

• fd: a connected socket descriptor (from connect or accept).
• buf / len: the bytes to send. len is a maximum; the return value is how many were actually accepted into the kernel send buffer.
• flags: bitwise-OR of MSG_*, or 0. Common: MSG_NOSIGNAL.
• Returns >= 0 bytes queued, or -1 with errno (EPIPE, ECONNRESET, EINTR, EAGAIN).
• A successful return does not mean the peer received the data — only that the kernel took it.

ssize_t recv(int fd, void *buf, size_t cap, int flags);

• cap: capacity of buf; the kernel writes at most this many bytes. It does not null-terminate — you do that yourself if treating data as a C string.
• Returns > 0 bytes read, 0 for orderly shutdown (EOF), or -1 with errno.

uint32_t htonl(uint32_t hostlong);   /* host -> network (big-endian) byte order */
uint32_t ntohl(uint32_t netlong);    /* network -> host byte order */

• Use these on any multi-byte integer you put on the wire so both ends agree on byte order regardless of CPU endianness. Declared in <arpa/inet.h>.

No heap is allocated by these calls, so there is nothing to free; you still own and must close(fd) when done.

Lesson

send() and recv(): the socket versions of write/read

send(fd, buf, n, flags) and recv(fd, buf, cap, flags) are the socket-specific siblings of write and read.

• For most code, pass 0 for flags.
• The flag bits (MSG_NOSIGNAL, MSG_DONTWAIT, ...) are situational and rarely needed at first.

The real challenge: short sends and short reads

The hard part of TCP is not the API. It is that a single call can move fewer bytes than you asked for. This is called a short send or read.

For example, send(fd, buf, 8, 0) might return 5. You then have to call again with the remaining buf + 5, 3.

The fix is to always loop until every byte is handled:

ssize_t send_all(int fd, const void *buf, size_t n) {
    const char *p = buf;
    while (n > 0) {
        ssize_t k = send(fd, p, n, 0);
        if (k < 0) { if (errno == EINTR) continue; return -1; }
        if (k == 0) return -1;
        p += k; n -= (size_t)k;
    }
    return 0;
}

What return values mean

• recv() returning 0 means the peer cleanly closed the connection (it sent a FIN). This is normal end-of-stream, not an error.
• recv() returning -1 (with errno set) is a real error.

Code examples

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <arpa/inet.h>   /* htonl / ntohl */

/* Write ALL n bytes, looping over short sends. Returns 0 on success, -1 on error. */
static int send_all(int fd, const void *buf, size_t n) {
    const char *p = buf;
    while (n > 0) {
        ssize_t k = send(fd, p, n, 0);
        if (k < 0) {
            if (errno == EINTR) continue;   /* interrupted: retry */
            return -1;                       /* real error */
        }
        if (k == 0) return -1;               /* shouldn't happen for send */
        p += k;
        n -= (size_t)k;
    }
    return 0;
}

/* Read EXACTLY n bytes, looping over short reads.
   Returns  1 = got all n, 0 = clean EOF before n, -1 = error. */
static int recv_all(int fd, void *buf, size_t n) {
    char *p = buf;
    while (n > 0) {
        ssize_t k = recv(fd, p, n, 0);
        if (k < 0) {
            if (errno == EINTR) continue;
            return -1;
        }
        if (k == 0) return 0;                /* peer closed early */
        p += k;
        n -= (size_t)k;
    }
    return 1;
}

/* Frame = 4-byte big-endian length prefix, then that many payload bytes. */
static int send_msg(int fd, const char *msg) {
    uint32_t len = (uint32_t)strlen(msg);
    uint32_t netlen = htonl(len);
    if (send_all(fd, &netlen, sizeof netlen) < 0) return -1;
    return send_all(fd, msg, len);
}

static int recv_msg(int fd, char *out, size_t cap) {
    uint32_t netlen;
    int r = recv_all(fd, &netlen, sizeof netlen);
    if (r <= 0) return r;                     /* 0 = EOF, -1 = error */
    uint32_t len = ntohl(netlen);
    if (len >= cap) return -1;                /* would overflow buffer */
    r = recv_all(fd, out, len);
    if (r != 1) return -1;                    /* short frame = protocol error */
    out[len] = '\0';
    return 1;
}

int main(void) {
    int sv[2];
    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0) { perror("socketpair"); return 1; }

    pid_t pid = fork();
    if (pid < 0) { perror("fork"); return 1; }

    if (pid == 0) {
        /* ---- child: the "server" that echoes framed messages ---- */
        close(sv[0]);
        char buf[256];
        for (;;) {
            int r = recv_msg(sv[1], buf, sizeof buf);
            if (r == 0) { printf("[server] peer closed (recv returned 0)\n"); break; }
            if (r < 0)  { printf("[server] error\n"); break; }
            printf("[server] got %zu bytes: \"%s\"\n", strlen(buf), buf);
            send_msg(sv[1], buf);            /* echo it back */
        }
        close(sv[1]);
        _exit(0);
    }

    /* ---- parent: the "client" ---- */
    close(sv[1]);
    const char *msgs[] = { "hello", "TCP is a byte stream", "goodbye" };
    char reply[256];
    for (int i = 0; i < 3; i++) {
        if (send_msg(sv[0], msgs[i]) < 0) { perror("send_msg"); break; }
        int r = recv_msg(sv[0], reply, sizeof reply);
        if (r == 1) printf("[client] echo: \"%s\"\n", reply);
        else        printf("[client] no reply (r=%d)\n", r);
    }
    close(sv[0]);                             /* sends FIN: server's recv now returns 0 */
    wait(NULL);
    printf("[client] done\n");
    return 0;
}

Line by line

• send_all — the core short-write loop. p walks forward and n counts down. If send returns -1 with errno == EINTR (a signal interrupted it) we simply retry; any other -1 is a genuine error. On each success we advance p += k and shrink n -= k. The loop ends only when every byte is queued.
• recv_all — the mirror image, with one extra case: recv == 0 means the peer closed the connection, so we return 0 to signal EOF rather than looping forever. Return 1 means we filled the whole request; -1 is an error.
• send_msg — implements framing. htonl converts the length to big-endian (network) byte order so the two ends agree regardless of CPU. We send_all the 4-byte prefix, then send_all the payload. Two send_alls, one logical message.
• recv_msg — reads the 4-byte prefix with recv_all; if that returns <= 0 we propagate EOF/error. ntohl converts back to host order. The bounds check if (len >= cap) return -1; is the security-critical line: it refuses a length that would overrun out before reading a single payload byte. Then recv_all(out, len) reads exactly the payload and we null-terminate so it prints as a string.
• socketpair — gives us two already-connected stream sockets without any real network, listener, or root. Perfect hermetic stand-in for a TCP connection.
• fork — the child plays server (echo loop), the parent plays client. Each closes the descriptor it does not use.
• The client loop sends three framed messages and reads each echo back. When it is done it closes its socket, which sends a FIN; the server's next recv_all returns 0, the server prints "peer closed" and exits. wait(NULL) reaps the child.

Common mistakes

1. Assuming one recv = one message

char buf[1024];
ssize_t n = recv(fd, buf, sizeof buf, 0);   // WRONG: assumes whole message arrives
parse(buf, n);

Why it breaks: TCP is a byte stream; recv may return a partial message or two messages glued together, so parse sees garbage or truncated data.

uint32_t len; recv_all(fd, &len, 4); len = ntohl(len);   // FIXED: frame + loop
recv_all(fd, buf, len);

2. Treating recv == 0 as an error

ssize_t n = recv(fd, buf, cap, 0);
if (n <= 0) { perror("recv"); exit(1); }   // WRONG: 0 is not an error

Why it breaks: 0 is the normal orderly-shutdown signal; this reports a spurious error (and perror prints a stale/meaningless errno) on every clean disconnect.

ssize_t n = recv(fd, buf, cap, 0);
if (n == 0)      { /* peer closed: clean up */ }
else if (n < 0)  { if (errno==EINTR) continue; perror("recv"); }
else             { /* got n bytes */ }

3. Trusting the length prefix from the wire

uint32_t len; recv_all(fd, &len, 4); len = ntohl(len);
char *p = malloc(len);          // WRONG: attacker sends len = 0xFFFFFFFF
recv_all(fd, p, len);

Why it breaks: an untrusted peer controls len; a huge value causes a giant allocation (DoS) or, with a fixed buffer, a heap/stack overflow.

if (len > MAX_MSG) return -1;   // FIXED: bound it before allocating/reading
char *p = malloc(len);

4. Ignoring SIGPIPE on send

send(fd, buf, n, 0);            // WRONG: peer gone -> SIGPIPE kills the process

Why it breaks: writing to a socket whose peer closed raises SIGPIPE, whose default action terminates the program — an easy remote crash.

send(fd, buf, n, MSG_NOSIGNAL); // FIXED: get EPIPE in errno instead of a signal

Debugging tips

• printf the return value and errno on every call during development: fprintf(stderr, "recv=%zd errno=%d (%s)\n", k, errno, strerror(errno));. Seeing a recv of 5 when you expected 8 instantly reveals a short read.
• strace -e trace=network ./prog (Linux) or dtruss (macOS) shows the exact send/recv syscalls and their byte counts — the ground truth for how the kernel split your stream.
• valgrind ./prog catches the classic framing bug where you read len bytes into a buffer smaller than len, or forget to null-terminate before printf("%s") (reads past the buffer).
• Watch for hangs: a recv_all that never returns usually means the peer sent fewer bytes than the length prefix promised (or you double-counted the prefix). Reproduce with a tiny sender that stops early.
• nc/ncat as a manual peer: nc -l 8080 on one side lets you type bytes and watch how your program frames and drains them.
• Under gdb, set a breakpoint on your recv_all loop and print n and k each iteration to confirm the pointer advances and the count decreases.

Memory safety

• recv never null-terminates. If you printf("%s", buf) a buffer that recv filled to capacity, you read past the end. Always reserve one byte and set buf[n] = '\0' yourself (as recv_msg does with out[len] = '\0').
• Bound every length that comes off the wire before using it to size a copy, index, or allocation. if (len >= cap) return -1; must run before recv_all(out, len). This single check is the boundary between a robust parser and a remote buffer overflow.
• Beware integer issues on the length prefix. A 32-bit len near SIZE_MAX, or arithmetic like malloc(len + 1) that wraps to 0, produces an undersized buffer. Compare against a fixed MAX_MSG and reject anything larger.
• send/recv return ssize_t (signed). Assigning to an int can truncate on huge buffers, and comparing an unsigned count against a signed return mixes signedness — keep the return in ssize_t and cast to size_t only after you have confirmed it is > 0.
• Concurrency: a socket fd shared between threads without synchronization can interleave partial sends from two threads, corrupting a frame. Give each connection to one thread, or hold a per-socket lock around a whole send_all.
• Always close(fd) when done; a leaked descriptor is both a resource leak and, for a server, a slow path to EMFILE denial of service.

Real-world uses

• Every TCP server and client: HTTP servers, database drivers (Postgres/MySQL wire protocols), message brokers, and RPC systems all sit on send/recv loops with framing on top.
• Length-prefix framing is exactly how protobuf-over-TCP, Thrift, and many custom binary protocols delimit messages; delimiter framing is how HTTP/1.1 and SMTP find message ends.
• Security tooling: IDS/IPS reassembly engines exist precisely because TCP boundaries are meaningless on the wire — an attacker can split a malicious payload across segments to evade naive inspectors, so defenders must reassemble the stream just like your recv_all does.
• Best practice: always use send_all/recv_all helpers (never a bare call), always frame your messages, always bound untrusted lengths, set MSG_NOSIGNAL on server sends, and treat recv == 0 as a normal lifecycle event. For high-concurrency servers, combine these with non-blocking sockets and an event loop (a later topic).

Practice tasks

1. Warm-up: Modify the example so the client sends a fourth message that is exactly 255 bytes long. Confirm it round-trips, then make it 256 bytes and observe/handle the len >= cap rejection.
2. Prove short reads exist: Write a sender that sends a 10-byte message in two send calls of 5 bytes each with a tiny usleep between them, and a receiver that logs the size of every recv. Show that a single recv sometimes returns fewer than 10 bytes.
3. Delimiter framing: Replace the 4-byte length prefix with newline-delimited messages. Write a recv_line that reads a byte at a time (or buffers) until it sees \n, and handle a line split across two recv calls.
4. Harden against a hostile peer: Add a MAX_MSG cap and a check that rejects a length prefix larger than it before reading the payload. Feed it a crafted 4-byte prefix of 0xFFFFFFFF and verify your program refuses it instead of crashing or hanging.
5. Echo server (capstone): Using the pattern from the echo-server exercise theme, build a program that loops recv-ing framed messages and send_alls them back, correctly terminating when recv returns 0, and never assuming a full message arrived in one call.

Summary

• send/recv are write/read plus a flags argument; pass 0 unless you need MSG_NOSIGNAL (server sends) or non-blocking behavior.
• TCP is a byte stream: any single call can move fewer bytes than requested, so always loop with send_all/recv_all.
• recv returning 0 = orderly shutdown (normal EOF); -1 = error (check errno, retry on EINTR); >0 = that many bytes, possibly a short read.
• Boundaries are erased on the wire — add your own framing (length prefix or delimiter).
• Bound any length taken from the wire before allocating or copying; trusting it is a remote overflow. Null-terminate before treating received bytes as a string, and close the fd when done.

Practice with these exercises

• Did we receive everything?
• Bytes left to send
• Send a message client → server (length-prefixed)
• Frame a length-prefixed message