TLS — read about it; never write it

Overview

You already know how to open a TCP connection with socket/connect and how to speak a plaintext line protocol like HTTP over it. TLS (Transport Layer Security) slots in between those two: your bytes go application → TLS → TCP on the way out, and TCP → TLS → application on the way in. Everything you learned about sockets still applies — TLS just wraps the file descriptor so that what actually travels on the wire is encrypted and tamper-evident.

This lesson is deliberately about reading TLS, not writing it. Cryptographic code is where subtle bugs become catastrophic vulnerabilities, so in practice you always wrap a vetted library — OpenSSL, BoringSSL, mbedTLS, or GnuTLS — and never roll your own. The goal here is that you can open a C file that uses OpenSSL, recognize the idioms, and know what each call is protecting you against. The hands-on demo parses TLS framing from a fixed buffer (no crypto, no network), which is exactly the kind of read-only structural understanding that makes someone else's TLS code legible.

Why it matters

Every HTTPS request, every git push over https, every modern message queue and database driver rides on TLS, so misreading it means misreading the security of nearly everything networked. The single most common real-world TLS bug is not a broken cipher — it is an application that encrypts the connection but forgets to verify the certificate, so an attacker on the path can present any certificate and silently man-in-the-middle the "secure" channel. Knowing where verification lives (and that OpenSSL leaves it OFF by default for clients) is the difference between a connection that is private and one that only looks private. Being able to read the record framing also lets you reason about downgrade attacks, truncation attacks, and why a length field must never be trusted before it is bounds-checked.

Core concepts

The three guarantees, and where each comes from

TLS adds three properties on top of a plain TCP stream:

Notice that authentication is a handshake job and the other two are record-layer jobs. A connection can be encrypted and still be worthless if authentication is skipped — encryption without authentication just means you are talking privately to the attacker.

The handshake: agree on secrets and identity

The handshake is the opening negotiation that happens once, before any application data flows. Conceptually:

Client                                  Server
  | --- ClientHello --------------------->  |   versions + cipher list + SNI + key share
  | <-- ServerHello ---------------------   |   chosen version + cipher + key share
  | <-- Certificate ---------------------   |   server proves who it is (signed by a CA)
  | <-- Finished ------------------------   |   "here is a MAC over everything so far"
  | --- Finished ------------------------>  |   both sides confirm they derived the same keys
  | ==== application_data (encrypted) ===>  |

The two sides use asymmetric cryptography (a public key anyone can see, a private key the server guards) only long enough to agree on a shared symmetric secret. From then on the fast symmetric key protects the bulk data. TLS 1.3 compresses this to a single round trip (1-RTT) by having the client guess the key-exchange parameters in the very first message.

Knowledge check: During the handshake, which single message is what actually lets the client believe the server is the real example.com and not an impostor who is also able to encrypt?

The Certificate message (validated against a trusted CA bundle, plus a hostname match). Encryption alone proves nothing about identity — an attacker can encrypt too. Only the certificate, chained to a CA you trust and matching the name you asked for, provides authentication.

The record layer: framing every byte after the handshake

Once keys exist, everything — remaining handshake messages, alerts, and your application data — is chopped into records. Every record starts with the same 5-byte header:

 byte:  0        1     2       3        4      5 ............ 5+len-1
      +--------+-----+-----+--------+--------+---------------------+
      | type   | ver_major | ver_minor | length (big-endian) | payload (len bytes) |
      +--------+-----+-----+--------+--------+---------------------+
         1 B        1 B       1 B        2 B                len B

The type byte tells you what the record carries; you can read it without any key:

The 2-byte length is the payload size that follows the header, capped at 2^14 (16384) for a plaintext record. That length is attacker-influenced data: a robust parser bounds-checks it against the bytes actually held before reading the payload — trusting a length blindly is a classic buffer-overread bug.

Certificate verification and SNI — the parts people get wrong

Two details cause most real incidents:

• Verification is opt-in for OpenSSL clients. If you never call SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL) and point at a real CA bundle, SSL_connect will happily complete against a forged certificate. You must also check the hostname (OpenSSL 1.1+: SSL_set1_host / X509_VERIFY_PARAM_set1_host) — a valid certificate for evil.com must not be accepted when you asked for example.com.
• SNI (Server Name Indication) is the hostname the client sends in the clear in the ClientHello so a server hosting many sites on one IP knows which certificate to present. In OpenSSL you set it with SSL_set_tlsext_host_name(ssl, "example.com"). Forgetting SNI often yields the wrong certificate and a confusing verification failure.

The defensive posture (and the auditor's view)

A safe client config is: SSL_VERIFY_PEER on, a real CA bundle loaded, hostname checked, and a minimum protocol of TLS 1.2 (SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION)) — refuse anything weaker. On the defensive/auditing side, misconfiguration shows up as expired or self-signed certificates, protocol downgrade to TLS 1.0/SSLv3, or weak cipher suites; defenders scan for these with testssl.sh, sslyze, and nmap --script ssl-enum-ciphers. All of that is lab-and-inspection work on your own or authorized hosts — the skill is hardening and verifying configuration, never attacking someone else's endpoint.

Syntax notes

OpenSSL's convention is unusual: most functions return 1 on success, not 0. The I/O calls (SSL_connect, SSL_read, SSL_write) return > 0 on success and <= 0 on failure, and you diagnose the failure with SSL_get_error.

SSL_CTX *SSL_CTX_new(const SSL_METHOD *method);
// Create a context (config template). method = TLS_client_method() or TLS_server_method().
// Returns NULL on failure. MUST be released with SSL_CTX_free.

void SSL_CTX_set_verify(SSL_CTX *ctx, int mode, SSL_verify_cb cb);
// mode = SSL_VERIFY_PEER to actually require+check the peer certificate.
// cb = NULL to use the default verifier. No return value.

int SSL_CTX_set_default_verify_paths(SSL_CTX *ctx);
// Load the system CA bundle. Returns 1 on success, 0 on failure.

int SSL_CTX_set_min_proto_version(SSL_CTX *ctx, int version);
// e.g. TLS1_2_VERSION. Returns 1 on success. Refuses older protocols.

SSL *SSL_new(SSL_CTX *ctx);
// One connection object from the context. NULL on failure. MUST be SSL_free'd.

int SSL_set_fd(SSL *ssl, int fd);          // bind an already-connected TCP fd; 1 = ok
int SSL_set1_host(SSL *ssl, const char *hostname);  // enable hostname verification; 1 = ok
long SSL_set_tlsext_host_name(SSL *ssl, const char *name); // set SNI (macro); 1 = ok

int SSL_connect(SSL *ssl);   // run the client handshake; 1 = ok, <=0 = check SSL_get_error
int SSL_write(SSL *ssl, const void *buf, int num);  // >0 = bytes written
int SSL_read (SSL *ssl, void *buf, int num);         // >0 = bytes read, 0 = closed
int SSL_get_error(const SSL *ssl, int ret);          // classify a <=0 return

int  SSL_shutdown(SSL *ssl);   // send close_notify; may need calling twice
void SSL_free(SSL *ssl);       // free the connection object
void SSL_CTX_free(SSL_CTX *ctx); // free the context (after all its SSLs are freed)

Must-free / must-close rules: every SSL_new pairs with SSL_free; every SSL_CTX_new pairs with SSL_CTX_free; the underlying TCP fd is still yours to close. Free the SSL objects before their SSL_CTX.

Lesson

TLS is the layer between TCP and your application. It provides three things:

• encryption
• authentication
• integrity

You will never write a TLS stack from scratch. Instead, you wrap a vetted library such as OpenSSL, BoringSSL, mbedTLS, or GnuTLS.

This lesson explains the structure of TLS so you can read someone else's TLS code with confidence.

Code examples

/* TLS record-layer parser (lab-only, fixed buffers, no crypto, no network).
 * Demonstrates the 5-byte TLS record header that frames every TLS message.
 * We NEVER implement TLS crypto here: we only *read* the framing so we can
 * reason about what a real library (OpenSSL, BoringSSL, ...) is doing.
 */
#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <stddef.h>

/* TLS ContentType values (record header byte 0). */
enum {
    CT_CHANGE_CIPHER_SPEC = 20,
    CT_ALERT              = 21,
    CT_HANDSHAKE          = 22,
    CT_APPLICATION_DATA   = 23,
};

static const char *content_type_name(uint8_t t) {
    switch (t) {
        case CT_CHANGE_CIPHER_SPEC: return "change_cipher_spec";
        case CT_ALERT:              return "alert";
        case CT_HANDSHAKE:          return "handshake";
        case CT_APPLICATION_DATA:   return "application_data";
        default:                    return "UNKNOWN";
    }
}

/* Map the legacy record version bytes to a human name. */
static const char *version_name(uint8_t major, uint8_t minor) {
    if (major != 3) return "non-TLS";
    switch (minor) {
        case 1: return "TLS 1.0";
        case 2: return "TLS 1.1";
        case 3: return "TLS 1.2"; /* TLS 1.3 still puts 0x0303 on the wire */
        default: return "TLS (unknown minor)";
    }
}

/* A parsed record header. length is payload bytes that FOLLOW the header. */
typedef struct {
    uint8_t  type;
    uint8_t  ver_major, ver_minor;
    uint16_t length;
} tls_record_hdr;

/* Parse one 5-byte header. Returns 0 on success, -1 if not enough bytes.
 * Bounds are checked BEFORE any read: never trust attacker-controlled lengths. */
static int parse_record_hdr(const uint8_t *buf, size_t avail, tls_record_hdr *out) {
    if (avail < 5) return -1;                 /* header itself is 5 bytes */
    out->type      = buf[0];
    out->ver_major = buf[1];
    out->ver_minor = buf[2];
    out->length    = (uint16_t)((buf[3] << 8) | buf[4]);  /* big-endian */
    return 0;
}

int main(void) {
    /* A tiny captured stream: two back-to-back TLS records living in one
     * fixed buffer. Bytes are hand-built, not received from any socket. */
    static const uint8_t stream[] = {
        /* Record 1: a Handshake record carrying a (truncated) ClientHello. */
        0x16,             /* content type = 22 (handshake)                 */
        0x03, 0x01,       /* record version = 0x0301 (legacy compat)       */
        0x00, 0x05,       /* length = 5 payload bytes                      */
        0x01,             /* HandshakeType = 1 (client_hello)              */
        0x00, 0x00, 0x01, /* Handshake length = 1 byte of body             */
        0x03,             /* body: legacy_version high byte, etc. (stub)   */

        /* Record 2: an Application Data record (ciphertext is opaque). */
        0x17,             /* content type = 23 (application_data)          */
        0x03, 0x03,       /* record version = 0x0303 (TLS 1.2/1.3)         */
        0x00, 0x04,       /* length = 4 payload bytes                      */
        0xDE, 0xAD, 0xBE, 0xEF   /* encrypted bytes: meaningless to us     */
    };

    size_t off = 0, avail = sizeof stream;
    int record_no = 0;

    while (off < avail) {
        tls_record_hdr h;
        if (parse_record_hdr(stream + off, avail - off, &h) != 0) {
            printf("truncated header at offset %zu\n", off);
            break;
        }
        /* Validate the declared length fits in what we actually hold. */
        size_t body_start = off + 5;
        if (h.length > avail - body_start) {
            printf("record %d claims %u payload bytes but only %zu remain\n",
                   record_no, h.length, avail - body_start);
            break;
        }

        printf("record %d: type=%-18s version=%-8s len=%u\n",
               record_no, content_type_name(h.type),
               version_name(h.ver_major, h.ver_minor), h.length);

        if (h.type == CT_HANDSHAKE && h.length >= 4) {
            uint8_t hs_type = stream[body_start];
            uint32_t hs_len = ((uint32_t)stream[body_start + 1] << 16)
                            | ((uint32_t)stream[body_start + 2] << 8)
                            |  (uint32_t)stream[body_start + 3];
            printf("   -> handshake msg type=%u len=%u%s\n",
                   hs_type, hs_len,
                   hs_type == 1 ? " (client_hello)" : "");
        } else if (h.type == CT_APPLICATION_DATA) {
            printf("   -> %u opaque ciphertext bytes (cannot inspect)\n", h.length);
        }

        off = body_start + h.length;   /* advance past this record */
        record_no++;
    }

    printf("parsed %d record(s), %zu bytes total\n", record_no, off);
    return 0;
}

Line by line

• enum { CT_... } + content_type_name: the record header's first byte is a ContentType. Naming the four values you'll actually see turns a raw 22 into handshake, which is exactly the read-only literacy this lesson is about.
• version_name: the legacy version field in the record header is 2 bytes (major.minor); 3.1=TLS 1.0 up to 3.3=TLS 1.2. TLS 1.3 keeps 0x0303 on the wire for compatibility and negotiates the real version inside the handshake — a subtlety the comment flags so you're not surprised.
• tls_record_hdr struct: models the 5-byte header exactly — 1 byte type, 2 bytes version, 2 bytes length. length is the count of payload bytes that follow, not including the header.
• parse_record_hdr: the safety heart of the program. It checks avail < 5 before touching any byte, so it can never read past the buffer. The length is reassembled big-endian (buf[3] << 8 | buf[4]) because TLS is a network protocol and network byte order is big-endian.
• stream[]: two hand-built records in one fixed array — a handshake record then an application_data record. Nothing here comes from a socket; it is a captured-bytes stand-in so the demo is hermetic.
• The while (off < avail) loop: walks record by record. body_start = off + 5 is where the payload begins.
• if (h.length > avail - body_start): the critical bounds check — the declared length is untrusted, so we confirm the payload actually fits in the bytes we hold before reading it. This is the exact defense against the buffer-overread class of TLS parser bugs.
• Handshake sub-parse: for a handshake record we peek one level deeper: the first payload byte is the HandshakeType (1 = client_hello) and the next 3 bytes are a 24-bit length. We print it to show that records contain structured messages.
• off = body_start + h.length: advance past this whole record to the next header, letting multiple records in one buffer be parsed in sequence.

Common mistakes

1. Encrypting but not verifying the certificate.

SSL_CTX *ctx = SSL_CTX_new(TLS_client_method());
SSL *ssl = SSL_new(ctx);
SSL_set_fd(ssl, fd);
SSL_connect(ssl);            // completes against ANY certificate!

Why it breaks: OpenSSL clients default to no verification, so a man-in-the-middle presenting a self-signed cert is accepted. The channel is encrypted — to the attacker.

SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
SSL_CTX_set_default_verify_paths(ctx);   // load system CA bundle
SSL_set1_host(ssl, "example.com");        // and check the hostname
if (SSL_connect(ssl) != 1) { /* handle handshake/verification failure */ }

2. Verifying the chain but not the hostname.

SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);   // chain OK...
// ...but no hostname check: a valid cert for evil.com is accepted for example.com

Why it breaks: a certificate can be perfectly valid and CA-signed yet issued for a different name. Without a hostname match, any valid cert works.

SSL_set1_host(ssl, "example.com");   // OpenSSL now rejects a name mismatch

3. Trusting the record/length field before bounds-checking it.

uint16_t len = (buf[3] << 8) | buf[4];
memcpy(payload, buf + 5, len);   // len came from the wire — overread if len > what we hold

Why it breaks: the length is attacker-controlled; copying len bytes from a short buffer reads out of bounds (heartbleed-shaped bug).

if (len > avail - 5) return -1;      // reject before reading
memcpy(payload, buf + 5, len);

4. Leaking OpenSSL objects / ignoring return codes.

SSL *ssl = SSL_new(ctx);
// ... use ssl, then just return;   // ssl and ctx leak; SSL_connect result ignored

Why it breaks: no SSL_free/SSL_CTX_free leaks memory and sockets; ignoring the != 1 return means a failed handshake looks like success.

if (SSL_connect(ssl) != 1) { /* diagnose with SSL_get_error */ }
SSL_shutdown(ssl); SSL_free(ssl); SSL_CTX_free(ctx);

Debugging tips

• openssl s_client -connect example.com:443 -servername example.com -showcerts is the gold-standard probe: it runs a full handshake, prints the certificate chain, negotiated protocol, and cipher, and reports the Verify return code. -servername sets SNI — omit it and you may get the wrong cert.
• Read the verify result explicitly. A non-zero Verify return code (e.g. 21 (unable to verify the first certificate), 10 (certificate has expired)) tells you exactly what failed. In code, SSL_get_verify_result(ssl) returns the same codes.
• SSL_get_error(ssl, ret) after a <= 0 I/O return classifies it: SSL_ERROR_WANT_READ/WRITE (retry, common with non-blocking fds), SSL_ERROR_SSL (protocol/verification error — pull details with ERR_get_error/ERR_error_string), SSL_ERROR_ZERO_RETURN (clean close).
• testssl.sh, sslyze, nmap --script ssl-enum-ciphers audit a deployment for weak protocols/ciphers and expired certs — run only against hosts you own or are authorized to test.
• For the framing demo: compile with -fsanitize=address,undefined and run under valgrind to prove the bounds checks hold; printf the offset and each header field to watch the parser advance record by record. A packet capture in Wireshark (filter tls) shows the same 5-byte headers on real traffic.

Memory safety

The dominant hazard in TLS-adjacent C is trusting length fields from the wire. Every length in a record header, handshake message, or extension is attacker-influenced; read even one byte of payload before confirming the declared length fits the bytes you actually hold and you have a buffer overread — the shape of Heartbleed. The demo's if (h.length > avail - body_start) check (and the avail - body_start form, which never underflows because body_start <= avail is already guaranteed) is the pattern to internalize: validate, then read.

With a real library the hazards shift to object lifetime and error handling. Free each SSL with SSL_free and each SSL_CTX with SSL_CTX_free, and free the SSL objects before the context they came from — using an SSL after its SSL_CTX is gone is use-after-free. Never ignore a <= 0 return from SSL_connect/SSL_read/SSL_write: treating a failed or partial operation as success can mean acting on unverified or truncated data. If you share one SSL_CTX across threads that is fine (it's a read-mostly template), but a single SSL connection object is not thread-safe — do not call SSL_read and SSL_write on the same SSL from two threads without your own synchronization.

Real-world uses

TLS underpins curl and every HTTPS client/server, git over https, database drivers (Postgres, MySQL), message brokers (Kafka, MQTT), gRPC, email (SMTP/IMAP over TLS), and effectively every modern protocol that needs authenticated encryption. Best-practice guidance: always start from a maintained wrapper (libcurl, or your language's stdlib TLS) rather than raw OpenSSL when you can; if you must use OpenSSL directly, turn on SSL_VERIFY_PEER, load a real CA bundle, set the hostname with SSL_set1_host, set SNI, and floor the protocol at TLS 1.2 (prefer 1.3). Keep the library patched — most historically severe TLS CVEs were memory-safety bugs in the parsing code, fixed by an update. On servers, disable legacy protocols and weak ciphers and rotate certificates before expiry; verify the result with testssl.sh or sslyze on your own infrastructure.

Practice tasks

1. Read a header by hand. Given the bytes 17 03 03 00 20, state the content type (name and number), the version, and how many payload bytes follow. Confirm by feeding equivalent bytes through the demo.
2. Add a record type. Extend content_type_name and the main loop to recognize an alert record (type 21) whose 2-byte payload is level then description; print, e.g., alert level=2 desc=40.
3. Harden the parser against a lie. Add a third record to stream[] whose declared length is larger than the remaining bytes, and confirm the bounds check catches it instead of overreading. Then rebuild with -fsanitize=address to prove no out-of-bounds read occurs.
4. Enforce a TLS 1.2 floor (reading exercise). Write the OpenSSL client setup that would reject anything below TLS 1.2, verify the peer, load the system CA bundle, and check the hostname example.com — as a commented sequence of calls (no network needed). Explain what each line defends against.
5. Frame a mock ClientHello (relates to tls-record-frame-parse). Build, in a fixed buffer, a valid handshake record whose payload is a HandshakeType 1 message with a correct 24-bit length, then have your parser walk it and print the message type and length. Deliberately corrupt the 24-bit length and confirm your code reports inconsistency rather than trusting it.

Summary

• TLS sits between TCP and your application and adds three things: confidentiality and integrity (record layer) plus authentication (handshake).
• The handshake negotiates version/cipher, proves identity with a CA-signed Certificate, and derives a shared symmetric key; TLS 1.3 does it in one round trip.
• After the handshake, every byte is framed in records with a 5-byte header: type (20/21/22/23), 2-byte version, 2-byte big-endian length. You can read the header without any key.
• Encryption without certificate + hostname verification is worthless — OpenSSL clients leave verification OFF by default, so you must turn on SSL_VERIFY_PEER, load a CA bundle, and call SSL_set1_host. Set SNI too.
• Never trust a length field from the wire before bounds-checking it; always SSL_free/SSL_CTX_free; check the 1/>0 return contract.
• Never implement TLS crypto yourself — wrap a vetted, patched library and configure it defensively (TLS 1.2 floor, verify peer). Audit with testssl.sh/sslyze on hosts you own.

Practice with these exercises

• Is the TLS version acceptable?
• Is it the HTTPS port?
• Parse the 5-byte TLS record header