What an operating system does

Overview

Every program you have ever run — a browser, a game, a compiler, a tiny C program with printf — runs on top of an operating system. The operating system (OS) is the software that sits between your programs and the physical hardware. It decides which program runs next, hands out memory, talks to disks and network cards, and enforces the rules about who is allowed to do what.

In the prerequisite lesson "CPU, RAM, disk, and processes" you saw the raw ingredients: a CPU that executes instructions, RAM that holds running data, disks that store files, and processes that are running programs. This lesson explains the manager that coordinates all of those ingredients so that dozens of programs can share one machine without trampling each other.

The heart of the OS is the kernel — the privileged, always-resident core. Most of the OS code is the kernel and the services around it. The single most important idea in this lesson is that there are two levels of trust on the machine:

• Kernel mode — full, unrestricted access to all hardware. The kernel runs here.
• User mode — a restricted sandbox. Your programs run here.

A program in user mode cannot reach the hardware directly. When it needs the OS to do something privileged — open a file, send network data, create a new process — it must make a system call, which is a controlled request that crosses from user mode into kernel mode and back. That controlled crossing is the gateway the whole rest of the course depends on, and it is also the place defenders watch most closely.

Why it matters

Understanding the OS is not optional background trivia — it is the model you will use every time you write or debug a program.

For everyday programming. When your C program calls fopen, read, or malloc, those eventually become system calls into the kernel. If you do not know that boundary exists, error messages like "Permission denied" or "Bad file descriptor" look like magic. Once you understand that the kernel is the thing saying "no," the errors make sense.

For performance. Crossing into the kernel has a cost. Knowing that a system call is more expensive than an ordinary function call explains why programs buffer I/O (one big read instead of a thousand tiny ones) and why excessive system calls slow a program down.

For security. The user/kernel boundary is the main security line on a host:

• Privilege escalation means a low-privileged user or process gaining higher privileges — for example, becoming root, or getting code to run in kernel mode.
• Kernel exploits are the most dangerous bugs, because the kernel is the highest authority; there is nothing above it to stop a compromised kernel.
• System calls are exactly where defenders watch for trouble, using tools such as auditd, seccomp (which can restrict the system calls a process may make), and EDR (Endpoint Detection and Response) agents.

Knowing where the boundary is — and that it is the boundary — is the foundation for everything you will later learn about permissions, processes, and defense.

Core concepts

This lesson has four core ideas. Learn each one on its own before connecting them.

1. The kernel

Definition. The kernel is the privileged core of the operating system — the part that is always loaded in memory and has complete control of the hardware.

Plain language. Think of the kernel as the building manager of a large shared office. Tenants (your programs) cannot rewire the electricity or open the server room themselves; they ask the manager, who has the master keys and decides what is allowed.

What it does internally. The kernel performs four big jobs:

• Scheduling — choosing which process gets the CPU next, and for how long, so many programs appear to run at once.
• Memory management — giving each process its own view of memory and stopping one process from reading or writing another's memory.
• Device control — talking to disks, keyboards, network cards, and screens through driver code.
• Security enforcement — checking permissions on every privileged action.

When it acts. The kernel runs in response to events: a system call, a hardware interrupt (e.g. "the disk finished reading"), or a timer that says "this process has had enough CPU; switch."

Pitfall. Beginners often picture the kernel as a program with its own window that is "running" alongside theirs. It is not. The kernel mostly sits idle and springs into action only when called or interrupted; it has no user interface of its own.

Knowledge check (explain in your own words): In one sentence, what is the difference between "the operating system" and "the kernel"?

2. User mode vs kernel mode

Definition. Modern CPUs support at least two privilege levels. Kernel mode allows every instruction and full hardware access. User mode forbids the dangerous instructions and restricts what memory and devices can be touched.

Plain language. It is the difference between an administrator account and a guest account — but enforced by the CPU hardware itself, not just by software rules.

How it works internally. The CPU keeps a flag (often called the current privilege level) that says which mode it is in. If user-mode code tries a privileged instruction — say, directly accessing a disk controller — the CPU refuses and traps into the kernel, which usually kills the offending process.

  +-------------------------------------------+
  |              USER MODE (low trust)        |
  |   your browser   your C program   a game  |
  |        |              |             |     |
  |        +------- system calls -------+     |
  +------------------------|------------------+
                           |  (controlled gate)
  +------------------------v------------------+
  |            KERNEL MODE (full trust)       |
  |   scheduler  memory mgr  drivers  security|
  |        |          |         |            |
  +--------|----------|---------|------------+
           v          v         v
         CPU        RAM      DEVICES (disk, NIC, ...)

When NOT to think you are in kernel mode. Almost all the code you write as an application programmer runs in user mode. You drop into the kernel only for the brief duration of a system call.

Pitfall. Assuming "my program is the administrator, so it can do anything." Even an administrator/root process still runs in user mode; being root raises what the kernel will permit, but the code still asks the kernel — it does not magically gain kernel-mode powers.

Knowledge check (concept): Why is the user/kernel split enforced by the CPU hardware rather than purely by the operating system software?

3. System calls

Definition. A system call is the controlled request a user-mode program makes to ask the kernel to perform a privileged operation on its behalf.

Plain language. It is the single official doorway between your program and the machine. Reading a file, writing to the screen, opening a network socket, creating a process — all go through this door.

How it works internally. When a program makes a system call:

1. It places a number identifying the requested operation (e.g. "read") and its arguments where the kernel expects them.
2. It executes a special instruction that switches the CPU into kernel mode and jumps to a fixed kernel entry point.
3. The kernel checks the request and permissions, does the work, and writes back a result.
4. The CPU switches back to user mode and the program continues.

  program        kernel
  (user mode)    (kernel mode)
     |
  read(fd,...)
     |---- switch to kernel mode --->|
     |                               | check fd is valid + allowed
     |                               | copy bytes from disk cache
     |<--- switch to user mode ------| return number of bytes read
     |
  continue

Structure. In C you rarely write the raw system-call instruction. You call a library function like read(), open(), or fork() from the C standard library, and it makes the system call for you. (The related exercises open-read-write-close and fork-basics use exactly these.)

When to use which. Use buffered, higher-level functions (fopen/fread) when you want convenience and efficiency; use the lower-level system-call wrappers (open/read) when you need fine control. Avoid making many tiny system calls in a loop — each one pays the cost of crossing the boundary.

Pitfall. Forgetting that a system call can fail. A read may return fewer bytes than requested, and open may fail with "Permission denied." Code that ignores the return value will misbehave.

Knowledge check (predict the behavior): A program calls read() 10,000 times, reading one byte each time, to load a file. A second program calls read() 10 times reading large chunks. Which is likely faster, and why?

4. The security boundary

Definition. The user/kernel transition (and, in user space, the line between an ordinary user and root) is the host's primary security boundary — the dividing line between different levels of trust.

Plain language. Crossing from a normal user to full control is the prize attackers want and the wall defenders guard.

How it works. Because every privileged action funnels through system calls, the kernel can check permissions in one place, and defenders can monitor those same calls. Privilege escalation is any technique that gets the kernel or a root process to perform an action on behalf of someone who should not be allowed.

When this matters. Whenever a program processes untrusted input that influences a privileged action (a file path, a command, a network message), the boundary is under pressure and must be defended with validation and least privilege.

Pitfall. Believing the boundary protects you automatically. It only protects you if privileged programs are written carefully; a buggy root program can be tricked into misusing its power.

Knowledge check (find the gap): If all privileged work goes through system calls, name one defensive tool from this lesson that watches those calls, and one thing it could detect.

Syntax notes

This is a concept lesson, so there is no new C syntax to memorize. What matters is recognizing which familiar library calls cross the kernel boundary. The annotated snippet below shows ordinary C function calls and notes where a system call happens underneath.

#include <stdio.h>   /* fopen, fread, fclose: buffered, fewer syscalls */
#include <fcntl.h>   /* open(): a direct system-call wrapper          */
#include <unistd.h>  /* read(), write(), close(): system-call wrappers */

/* High-level: the C library buffers and makes syscalls for you. */
FILE *f = fopen("data.txt", "r");   /* underneath: an open() syscall   */

/* Low-level: each call here is (essentially) one system call. */
int fd = open("data.txt", O_RDONLY); /* asks kernel to open the file   */
char buf[256];
ssize_t n = read(fd, buf, sizeof buf); /* asks kernel for bytes        */
close(fd);                            /* asks kernel to release the fd  */

Key points the syntax hints at:

• A file descriptor (fd) is a small integer the kernel hands back to identify an open resource. You pass it to later system calls.
• read returns the number of bytes actually read (ssize_t), which can be less than you asked for, or -1 on error.
• Higher-level (fopen/fread) means fewer trips across the boundary; lower-level (open/read) means more control.

Lesson

The operating system (OS) is the software that manages the hardware. It also decides which programs may access what, and when.

The kernel

The kernel is the privileged core of the OS. It does the most sensitive work:

• schedules processes (decides which program runs next)
• manages memory
• controls devices
• enforces security

The kernel runs in kernel mode, with full hardware access. Your programs run in user mode, with restricted access.

System calls

A user program cannot touch the hardware directly. Instead, it asks the kernel to act on its behalf through a system call — for example, reading a file, opening a network socket, or creating a process.

During a system call, the CPU switches from user mode to kernel mode, runs the requested work, then switches back. System calls are the one controlled gateway between an application and the machine.

Why this boundary matters

The user/kernel split is the primary security boundary on a host.

• Privilege escalation usually means tricking a kernel-mode or root component into doing something for the attacker.
• Kernel exploits are the most powerful, because the kernel has no higher authority above it to step in and stop it.

Common OS families

Unix-like systems (Linux, macOS, BSD) and Windows differ in the details, but they share the same core ideas:

• a kernel
• processes
• users
• permissions
• system calls

The labs in this course are Unix/Linux-flavoured.

Code examples

Below is a complete, compilable C11 program that makes the user/kernel boundary visible. It opens a file with the low-level open system-call wrapper, reads it in chunks with read, prints how many bytes each call returned, and cleans up. Watching the byte counts shows that each read is a separate request to the kernel.

#include <stdio.h>      /* printf, perror */
#include <fcntl.h>      /* open, O_RDONLY */
#include <unistd.h>     /* read, close */
#include <errno.h>      /* errno */

int main(int argc, char *argv[]) {
    if (argc != 2) {
        fprintf(stderr, "usage: %s <filename>\n", argv[0]);
        return 1;
    }

    /* Ask the kernel (via a system call) to open the file read-only. */
    int fd = open(argv[1], O_RDONLY);
    if (fd == -1) {                 /* the kernel refused or file is missing */
        perror("open");             /* prints the kernel's reason, e.g. ENOENT */
        return 1;
    }

    char buf[64];                   /* small buffer to force several reads */
    ssize_t n;
    long total = 0;
    int calls = 0;

    /* Each loop iteration is one read() system call into the kernel. */
    while ((n = read(fd, buf, sizeof buf)) > 0) {
        calls++;
        total += n;
        printf("read() call #%d returned %zd bytes\n", calls, n);
    }

    if (n == -1) {                  /* read can fail mid-way */
        perror("read");
        close(fd);                  /* still release the descriptor */
        return 1;
    }

    printf("done: %d system calls, %ld bytes total\n", calls, total);

    if (close(fd) == -1) {          /* releasing the fd is also a syscall */
        perror("close");
        return 1;
    }
    return 0;
}

What it does. Given a filename, the program asks the kernel to open it, then repeatedly asks the kernel for up to 64 bytes at a time until the file ends. It counts how many read system calls were needed and the total bytes.

Expected output. For a file containing the 13 bytes hello, world\n, compiled as gcc -std=c11 -Wall -o readfile readfile.c and run as ./readfile hello.txt:

read() call #1 returned 13 bytes
read() call #2 returned 0 bytes   (loop exits; 0 means end of file)
done: 1 system calls, 13 bytes total

A larger file would show several read() lines, each one a separate trip across the user/kernel boundary.

Key edge cases.

• Missing file or no permission: open returns -1 and perror prints the kernel's reason (e.g. No such file or directory or Permission denied). This is the security boundary saying "no."
• Short reads: read may legitimately return fewer bytes than the buffer size even when the file is not finished; the loop handles this by looping until read returns 0.
• End of file: read returns 0, which ends the loop. A return of -1 means a real error, which is why it is checked separately.

Line by line

Here is what happens when you run ./readfile hello.txt on a 13-byte file.

1. argc != 2 check — confirms exactly one filename was given. If not, it prints usage and exits. This is input validation before doing any work.
2. open(argv[1], O_RDONLY) — the program leaves user mode for a moment: the C library executes the open system call. The kernel checks the path and your permissions, then returns a file descriptor (a small integer such as 3) into fd. Control returns to user mode.
3. if (fd == -1) — if the kernel refused (file missing, no read permission), open returned -1. perror("open") prints the kernel's reason. This is the boundary visibly denying access.
4. The while loop — each read(fd, buf, 64) is another crossing into kernel mode. The kernel copies up to 64 bytes from its disk cache into buf and returns the count in n.
5. First iteration — for a 13-byte file, read returns 13; calls becomes 1, total becomes 13, and the program prints the line.
6. Second iteration — the file is now fully read, so read returns 0. The loop condition > 0 is false, so the loop ends without incrementing calls.
7. if (n == -1) — n is 0 here, not -1, so this error branch is skipped. (If a disk error had occurred mid-read, n would be -1 and this branch would run.)
8. Summary printf — prints done: 1 system calls, 13 bytes total.
9. close(fd) — one final system call asks the kernel to release the descriptor so the resource is not leaked. Even a missing close would be cleaned up at exit, but explicit cleanup is the correct habit.

A small trace of the loop variables for the 13-byte file:

iteration | read() returns n | calls | total | action
----------+------------------+-------+-------+-----------------------
   1      |        13        |   1   |  13   | print, continue
   2      |         0        |   1   |  13   | loop exits (EOF)

The single takeaway: each open, read, and close you see is a controlled request that briefly enters the kernel and comes back.

Common mistakes

Mistake 1: Ignoring the return value of read.

/* WRONG: assumes read always fills the whole buffer */
read(fd, buf, sizeof buf);
process(buf, sizeof buf);   /* may process leftover/garbage bytes */

Why it is wrong: read can return fewer bytes than requested (a short read), 0 at end of file, or -1 on error. Treating the buffer as full leads to processing stale or uninitialized data.

/* CORRECT: use the actual count, and check for error/EOF */
ssize_t n = read(fd, buf, sizeof buf);
if (n == -1) { perror("read"); return 1; }
if (n == 0) { /* end of file */ }
process(buf, (size_t)n);

How to recognize it: intermittent garbage in output, or bugs that depend on file size. Always look at what read returned.

Mistake 2: Thinking root (or "administrator") means kernel mode.

Learners assume that running as root lets their code do anything directly. In reality, a root process still runs in user mode and still makes system calls; being root only raises what the kernel will permit. The fix is the mental model: privilege (who you are) and CPU mode (where the code runs) are two different things.

Mistake 3: Forgetting to close descriptors.

/* WRONG: file descriptor leaked in the error path */
int fd = open(path, O_RDONLY);
if (something_failed) return 1;   /* fd never closed */

Why it is wrong: open descriptors are a limited kernel resource. A long-running program that leaks them eventually hits Too many open files. The fix is to close(fd) on every exit path (as the lesson's code does even in the read-error branch). Recognize it by watching the descriptor count of a long-running process grow over time.

Mistake 4: Making thousands of tiny system calls.

Reading one byte at a time in a loop is correct but slow, because every byte pays the cost of crossing into the kernel. Recognize it from poor performance on large files; fix it by reading in larger chunks (or using buffered fread).

Debugging tips

Compiler errors.

• implicit declaration of function 'read'/'open'/'close' — you forgot #include <unistd.h> (for read/close) or #include <fcntl.h> (for open). Add the headers.
• 'O_RDONLY' undeclared — same fix: include <fcntl.h>.
• Format-string warnings on %zd/%zu — ssize_t/size_t use %zd/%zu; mismatched specifiers cause warnings. Compile with -Wall so you see them.

Runtime errors (the kernel telling you something).

• No such file or directory (ENOENT) — wrong path or the file does not exist. Check with ls.
• Permission denied (EACCES) — the kernel's security check failed; you lack read permission. This is the security boundary doing its job.
• Too many open files (EMFILE) — you are leaking descriptors; find the missing close.

Logic errors.

• Loop never ends, or reads forever — you may be using != against the wrong value, or ignoring the 0 (EOF) return. The loop condition should stop on <= 0.
• Truncated output — you printed sizeof buf bytes instead of the n bytes actually read.

Concrete debugging steps.

1. Always print or check the return value of every system call, plus perror to see the kernel's reason.
2. On Linux, run your program under strace ./readfile hello.txt to see every system call it makes, with arguments and results. This makes the user/kernel boundary literally visible and is the single best tool for this topic. (macOS has dtruss.)
3. Ask yourself: Did the call succeed? What did it return? What does the kernel say the error is?

Memory safety

This is a non-security, concept lesson, so the focus here is robustness and correct use of the boundary rather than attack/defense. Still, the way you cross into the kernel has real safety implications.

Buffer bounds. Always pass the true buffer size to read (sizeof buf), and only treat the returned count as valid data. Writing past the buffer, or using bytes beyond what read returned, is undefined behavior and a classic source of crashes and vulnerabilities.

Uninitialized data. Bytes in buf beyond the returned count are unspecified. Never print or process them as if they were file contents.

Resource lifetime. File descriptors are kernel resources with a finite limit. Close every descriptor you open, on every code path, so the kernel can reclaim them.

Integer/overflow care. When accumulating totals (the long total in the example), be mindful that very large files can exceed a type's range; choose a wide enough type for the data you expect.

Trusting return values. Treat -1 as a real error every time and inspect errno (via perror). Silently ignoring a failed system call is how small bugs become security bugs in privileged programs.

Authorization note. Everything in this course's labs runs on your own local machine or in an isolated container. Never use system-call tracing or file access against systems you are not authorized to inspect.

Real-world uses

Where these ideas show up.

• Every server and desktop OS. Linux powers most web servers, Android phones, and cloud infrastructure; macOS and Windows run on laptops and workstations. All of them rely on the same kernel/user-mode/system-call model from this lesson.
• Containers (Docker, Kubernetes). Containers share one host kernel but isolate processes; tools like seccomp restrict which system calls a container may make, directly applying the boundary you learned here.
• Security monitoring. Defensive tooling — auditd, EDR agents, and sandboxes — works by observing system calls, because that is the choke point where all privileged actions pass.
• Performance engineering. Databases and high-throughput servers minimize system calls (batching, buffering, memory-mapped I/O) precisely because crossing the boundary has a cost.

Professional best-practice habits for this topic.

Beginner rules:

• Always check the return value of open, read, write, and close, and use perror/errno to understand failures.
• Close every descriptor you open, on every path.
• Use meaningful names (fd, bytes_read) and comment only the non-obvious lines.
• Prefer buffered I/O (fopen/fread) for ordinary file reading unless you need low-level control.

Advanced habits:

• Handle short reads and EINTR (a system call interrupted by a signal) by retrying or looping correctly.
• Apply least privilege: drop elevated privileges as early as possible, and restrict allowed system calls with seccomp where appropriate.
• Profile system-call counts (e.g. with strace -c) to find performance hot spots.
• Validate any untrusted input that feeds into a privileged system call (paths, sizes), because privileged programs are where the security boundary is most tested.

Practice tasks

Beginner 1 — Name the boundary. Objective: Solidify the mental model. In your own words (3–5 sentences), explain what happens, step by step, when a user program calls read() to get data from a file. Requirements: mention user mode, kernel mode, the system call, and the file descriptor. Hint: use the data-flow diagram from Core Concepts. Concepts: system calls, user/kernel mode.

Beginner 2 — Observe a failure. Objective: See the security boundary respond. Compile the lesson's readfile program and run it on (a) a file that exists and you can read, and (b) a path that does not exist. Record the exact message perror prints in each case. Expected: the second run prints something like open: No such file or directory. Constraint: do not modify the program. Hint: invent a clearly fake filename like nope.txt. Concepts: system-call failure, errno/perror.

Intermediate 1 — Count the crossings. Objective: Connect buffer size to system-call count. Modify readfile so the buffer size comes from a second command-line argument. Run it on the same medium-sized file with buffer sizes 16, 256, and 4096, and report how many read() calls each took. Input/output example: ./readfile big.txt 256 prints the per-call lines and a final count. Constraint: validate the argument is a positive integer. Hint: a bigger buffer means fewer crossings. Concepts: system calls, performance, input validation.

Intermediate 2 — Safe copy. Objective: Use open/read/write/close together correctly. Write a program mycp src dst that copies a file by reading chunks and writing them out. Requirements: check every return value, handle short reads, close both descriptors on every path, and report the total bytes copied. Constraint: do not assume read fills the buffer. Hint: loop on read until it returns 0, and within each chunk loop on write until all n bytes are written. Concepts: system calls, resource cleanup, short reads/writes.

Challenge — Trace and explain. Objective: Make the boundary visible and analyze it. On Linux, run strace ./readfile hello.txt (or strace -c) and study the output. Deliver a short write-up that: lists the system calls the program made in order, explains what each one did, identifies which calls crossed into the kernel, and explains why the counted read() calls in your program's output match (or differ from) the read lines in the strace output. Constraint: run only on your own machine or an authorized lab/container. Hint: compare strace's view with your program's printed counts. Concepts: system calls, debugging with tracing, the user/kernel boundary.

Summary

• The operating system manages the hardware and shares it safely among many programs; its privileged core is the kernel, which handles scheduling, memory, devices, and security.
• The CPU enforces two trust levels: kernel mode (full hardware access) and user mode (restricted). Your programs run in user mode; the kernel runs in kernel mode.
• Programs cross that boundary only through system calls — controlled requests like open, read, write, and close. In C you usually invoke them via library wrappers, and each crossing has a cost.
• The most important syntax habit: treat every system call's return value as meaningful — -1 is an error (check perror/errno), 0 from read is end of file, and the returned count is the only valid amount of data.
• Common mistakes: ignoring read's return value, confusing root with kernel mode, leaking file descriptors, and making too many tiny system calls.
• The user/kernel line is the host's central security boundary: privilege escalation is about abusing it, and defenders monitor system calls to watch it. Everything you learn next about files, permissions, and processes builds on this boundary.

Practice with these exercises

• Interpret fork's return
• Processes after N forks
• Read with raw descriptors
• Write with raw descriptors