Linux System Programming · intermediate · ~15 min

rlimit — capping CPU, memory, fds

Learn to set resource limits on your own process before escalating privileges or entering a sandbox.

Overview

setrlimit puts a per-process cap on a single resource, such as memory or CPU time.

Each resource has two limits:

  • The soft limit is the value currently enforced.
  • The hard limit is the ceiling for the soft limit. A normal process may raise its soft limit only up to the hard limit.

Once you lower the hard limit, only root can raise it again. This is a one-way door, which makes it useful for dropping your own privileges before running risky code.

Why it matters

Sandboxing in C starts with resource limits.

Every Docker container, every CI runner, and every code-execution service tunes them to keep workloads contained.

Core concepts

Common resources you can cap

  • RLIMIT_AS — total address space (anonymous memory plus mmap'd regions). This effectively caps how much malloc can hand out.
  • RLIMIT_CPU — CPU seconds consumed. The kernel sends SIGXCPU when the soft limit is reached, and SIGKILL at the hard limit.
  • RLIMIT_FSIZE — maximum size of a file the process may write. Overrunning it raises SIGXFSZ.
  • RLIMIT_NOFILE — maximum number of open file descriptors.
  • RLIMIT_NPROC — maximum number of processes for the user.

Pentester mindset

When auditing a sandbox runner, check whether it sets resource limits at all.

  • Missing RLIMIT_CPU means an attacker can pin a CPU core forever.
  • Missing RLIMIT_AS means an attacker can exhaust memory and trigger the out-of-memory (OOM) killer.

Defensive coding habit

Lower both the soft and hard limits at the start of a section that runs with privileges before dropping them.

Never raise a hard limit that you set yourself. You cannot, unless you are root.

Syntax notes

#include <sys/resource.h>

int getrlimit(int resource, struct rlimit *rlim);
int setrlimit(int resource, const struct rlimit *rlim);

getrlimit reads the current limits for a resource. setrlimit changes them.

Lesson

setrlimit and getrlimit configure per-process resource caps: CPU seconds, address space, file size, and number of open file descriptors.

They are the user-space foundation of every sandbox, including ours.

Code examples

struct rlimit r = { .rlim_cur = 128 * 1024 * 1024, .rlim_max = 128 * 1024 * 1024 };
setrlimit(RLIMIT_AS, &r);     /* 128 MB max address space */

Line by line

struct rlimit r;
r.rlim_cur = r.rlim_max = 5;              /* 5 CPU-seconds */
setrlimit(RLIMIT_CPU, &r);
r.rlim_cur = r.rlim_max = 128 * 1024L * 1024L;   /* 128 MB */
setrlimit(RLIMIT_AS, &r);

Common mistakes

  • Raising a hard limit after dropping it. Once you lower a hard limit, you cannot raise it again unless you are root.

Debugging tips

Several tools let you inspect resource limits:

  • ulimit -a in a shell shows your current limits.
  • prlimit --pid PID queries any running process.
  • cat /proc/PID/limits shows the kernel's view for a given process.

Memory safety

Setting RLIMIT_AS too low can make a later malloc fail in ways you did not expect.

Always test under realistic load before relying on a tight limit.

Real-world uses

Concrete uses: Resource limits are how a system stops one process from taking everything. A shell sets them via ulimit before launching a job; a container runtime applies them per workload; a build server caps memory so a runaway compile cannot swap the machine to a halt. RLIMIT_NOFILE is the limit servers raise before accepting thousands of connections, and RLIMIT_CORE decides whether a crash produces a core dump for debugging. Sandboxes — including the one grading these exercises — set CPU, memory and process limits so untrusted code cannot exhaust the host.

Professional best practices:

Beginner:

  • Read a limit with getrlimit before assuming what it is; defaults differ widely between systems.
  • Remember the distinction between the soft limit (what applies now) and the hard limit (the ceiling the soft limit may be raised to).

Intermediate:

  • An unprivileged process can lower a hard limit but never raise it — plan the order of operations accordingly, and note the change is irreversible for that process.
  • Set limits before dropping privileges, and check the return value; a silently failed setrlimit leaves the process unprotected.
  • Prefer cgroups on Linux when you need limits applied to a whole process tree rather than a single process.

Practice tasks

  1. Cap RLIMIT_AS to 128 MB, then watch a 200 MB malloc fail.
  2. Cap RLIMIT_CPU to 5 seconds, then run an infinite loop and observe the process being killed.
  3. Lower RLIMIT_NOFILE to 16, then watch the 17th open call fail.

Summary

  • setrlimit caps a per-process resource (CPU, memory, file size, open fds).
  • Each resource has a soft limit (enforced) and a hard limit (the ceiling).
  • Lowering the hard limit is permanent for a non-root process, so use it to drop privileges.
  • Resource limits are the kernel-backed foundation of every sandbox.

Practice with these exercises