Cloud & Container Security · beginner · ~11 min
**What you will learn** - Explain the difference between a container and a virtual machine (VM), and why containers share the host kernel. - Distinguish a Docker **image** (read-only layered template) from a **container** (a running instance) and read the layers with `docker history`. - Identify the five most common image/Dockerfile security risks: baked-in secrets, untrusted or unpinned base images, running as root, build-arg leaks, and oversized images full of CVEs. - Write a hardened Dockerfile: pinned base image by digest, non-root `USER`, no secrets in layers, and a minimal final image. - Inspect an existing image to detect a leaked secret, and verify your fix in a local lab. - Apply runtime hardening flags (`--read-only`, `--cap-drop`, non-root) when you run a container.
A container is a way to package an application together with everything it needs to run — the code, the runtime, the libraries, and the configuration — so it behaves the same on your laptop, in continuous integration (CI), and in production. Before containers, "it works on my machine" was a real and painful problem: the code depended on a specific library version or environment variable that existed on the developer's computer but not on the server. Containers solve this by shipping the environment along with the app.
The key thing that makes containers lightweight is also what makes them less isolated than a VM. A virtual machine boots a whole guest operating system with its own kernel on top of a hypervisor. A container does not — it shares the host's kernel and is really just a set of isolated processes, fenced off using Linux kernel features (namespaces and cgroups). Starting a VM takes seconds to minutes and gigabytes of memory; starting a container takes milliseconds and megabytes. The trade-off is that the isolation boundary between a container and the host is thinner, so a flaw in the kernel or a misconfiguration can let an attacker "break out" of the container onto the host.
Two terms you must keep straight:
This is exactly like a class and an object, or a program file on disk versus a running process. One image can produce many containers.
Most real-world container security problems are not exotic kernel exploits — they are image-build hygiene mistakes that anyone can find and fix: a secret accidentally copied into a layer, a latest base image from an unknown source, or a container left running as root. This lesson teaches you to spot and fix those. It leads directly into the next lesson, Container runtime risks and Kubernetes, which covers what happens once these images are deployed and orchestrated at scale.
Containers are everywhere: web backends, databases, CI pipelines, machine-learning jobs, and the building blocks of every Kubernetes cluster. Because they are built from layered images that get pushed to shared registries and pulled by many machines, a single bad image can spread a vulnerability across an entire fleet.
Three concrete reasons this matters:
FROM someimage:latest, you are trusting whoever published that image and re-pulling a moving target. If their account is hijacked, your build silently ships malware.The encouraging part: these are quick to find (often by reading the Dockerfile and running one inspection command) and quick to fix. Image hygiene gives a very high security return for very little effort.
Definition. An image is an immutable, read-only template built from stacked layers. A container is a live, running instance of an image with a thin writable layer added on top.
How it works internally. Each instruction in a Dockerfile (FROM, COPY, RUN, …) usually creates a new layer — a snapshot of the filesystem changes that instruction made. Layers are stacked using a union filesystem, so the final image is the sum of all layers viewed as one filesystem. Layers are cached and shared between images, which is why builds are fast and images are compact. When you run a container, Docker adds one writable layer where the running process can make changes; everything below stays read-only and shared.
CONTAINER (running) IMAGE (read-only template)
+-------------------------+
| writable layer (R/W) | <-- per-container changes live here
+=========================+ <== everything below is shared, read-only
| layer 4: COPY app . |
| layer 3: RUN pip install|
| layer 2: COPY reqs.txt |
| layer 1: FROM python | (base image layers)
+-------------------------+
When to use / not. Build an image once per version; spin up containers as many times as you need. Do not store important data only in a container's writable layer — it vanishes when the container is removed. Use a volume for persistent data.
Pitfall. People assume "if I rm the file in a later step, it's gone." It is not — the earlier layer that added the file is still in the image.
Knowledge check: In your own words, why can one image produce many running containers, and where do per-container changes go?
Definition. A baked-in secret is any credential (API key, password, token, .env file, private key) that ends up stored inside an image layer.
How it works internally. Because every layer is preserved, a secret added by COPY .env . or ENV API_KEY=... is written into a layer permanently. Even this fails to remove it:
Layer A: COPY secret.txt /app/secret.txt <-- secret recorded HERE, forever
Layer B: RUN rm /app/secret.txt <-- only hides it in the FINAL view
Layer B makes the file invisible when you ls inside the running container, but Layer A still contains the bytes. Anyone can run docker history to see the commands and extract the layer's contents.
When to use / not. Never bake secrets in — there is no safe way to do it. Instead inject secrets at runtime (environment variables, mounted files, or a secrets manager) or use Docker build secrets (--mount=type=secret) that are available only during a build step and not stored in a layer.
Pitfall. Thinking a private repo or a private registry makes baked secrets safe. Anyone who pulls the image — or any leaked image tarball — still gets the secret.
Knowledge check (find-the-bug): A teammate says, "I copied the AWS key in, used it during the build, then deleted it in the next
RUN, so the image is clean." What is wrong with this reasoning?
Definition. The base image is what your FROM line starts from. Everything you add sits on top of it.
How it works internally. A large base image (a full OS) bundles hundreds of packages you do not use, each of which may carry a known vulnerability (a CVE — a publicly catalogued flaw). A minimal base (such as a slim or distroless image that contains only your runtime, no shell or package manager) has a far smaller attack surface. A pinned base — referenced by an exact version and ideally a content digest (sha256:...) — guarantees you rebuild the same bytes every time. latest is a moving tag that can change under you.
When to use / not. Use the smallest base that still runs your app. Pin to a digest for production builds. Avoid unknown publishers; prefer official or verified images, and scan whatever you pull.
Pitfall. FROM ubuntu:latest looks convenient but is unpinned (non-reproducible), large (more CVEs), and a moving supply-chain target.
Knowledge check: Why is pinning
FROM python:3.12-slim@sha256:abc...safer thanFROM python:latest?
USERDefinition. The USER instruction sets which OS user the container's process runs as. By default it is root (UID 0).
How it works internally. A container shares the host kernel. If a process running as root inside the container escapes its isolation (via a kernel bug, a mounted Docker socket, or excessive capabilities), it can act as root on the host. Running as an unprivileged user means an escape lands you as a low-privilege user, dramatically reducing the damage.
Trust boundary: [ container process ] --| kernel |-- [ host ]
root inside + breakout -> root on host (catastrophic)
nonroot inside + breakout -> unprivileged on host (contained)
When to use / not. Create and switch to a non-root user in almost every image. The rare exception is a process that genuinely needs a privileged operation — and even then, drop every capability you do not need.
Pitfall. Adding USER but having earlier steps create files owned by root that the app then cannot write — set ownership (chown) when you copy app files.
Knowledge check (predict-the-output): A container runs as root and someone exploits a flaw to break out to the host. How does this differ from the same breakout in a container that ran as a non-root user?
Definition. A build-arg leak is a secret passed via ARG (or --build-arg) that ends up recorded in the image's build metadata or layers.
How it works internally. ARG values are build-time variables. They can be captured in the image history and, if referenced into an ENV or a RUN command, written into a layer. So --build-arg API_KEY=... is not a safe way to inject secrets.
When to use / not. Use ARG only for non-sensitive build configuration (a version number, a feature flag). For secrets during a build, use BuildKit build secrets (RUN --mount=type=secret), which are mounted only for that step and never persisted.
Pitfall. Treating ARG like a secure vault. It is not — it is convenience configuration, often visible via docker history.
Knowledge check: Name one safe thing to pass via
ARGand one thing you must never pass viaARG.
A Dockerfile is a plain-text recipe. The most security-relevant instructions, annotated:
# Pin the base image to a version AND a digest for reproducibility + trust
FROM python:3.12-slim@sha256:<digest>
# ARG = build-time config (NON-secret only); may appear in image history
ARG APP_VERSION=1.0.0
# Create a dedicated unprivileged user (no login shell, no home clutter)
RUN useradd --system --no-create-home --shell /usr/sbin/nologin appuser
WORKDIR /app
# Copy dependency manifest first so this layer caches independently of code
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy the app and hand ownership to the non-root user in one step
COPY --chown=appuser:appuser . .
# Build-time secret: mounted ONLY for this RUN, never written to a layer
RUN --mount=type=secret,id=build_token \
sh -c 'use_token "$(cat /run/secrets/build_token)"'
# Drop privileges BEFORE the process runs
USER appuser
# Document the listening port (metadata only; does not publish it)
EXPOSE 8000
# Run the app (exec form so signals reach the process correctly)
CMD ["python", "app.py"]
Key rules:
ARG is build-time and not secret-safe; ENV persists into the image (never put secrets there).COPY --chown sets file ownership so a non-root USER can actually use the files.RUN --mount=type=secret (BuildKit) is the supported way to use a secret during a build without baking it in.USER before CMD/ENTRYPOINT so the workload runs unprivileged.Containers bundle an application with its dependencies and run it as isolated processes that share the host kernel. This makes them lighter than VMs, but the isolation boundary is thinner.
An image is a read-only template made of layers. A container is a running instance of that image. Images come from registries such as Docker Hub or a private registry.
ENV or COPY of a key or .env file stays in the image layers, even if you "remove" it in a later step. Anyone who pulls the image can extract it. Use build secrets or runtime injection instead. Never bake secrets in.USER.latest from an unknown source is a supply-chain risk. Pin image digests and use trusted bases.ARG can persist in the build history.docker history or by extracting layers) to find secrets and sensitive files.Most container findings come down to image-build hygiene.
Below is a complete, realistic comparison: an insecure Dockerfile, then a secure rewrite, plus the commands to build and verify the fix in a local lab.
WARNING: Intentionally vulnerable training example — use only in a local, isolated, authorized lab. Do not deploy.
# insecure.Dockerfile -- DO NOT DEPLOY
FROM python:latest # unpinned, huge, moving target
WORKDIR /app
COPY . . # copies the whole repo, incl. .env / keys
ENV API_KEY=sk-live-1234567890abcdef # secret baked into a layer FOREVER
COPY secret.pem /app/secret.pem
RUN rm /app/secret.pem # 'deleted' but still in an earlier layer
RUN pip install -r requirements.txt
# no USER -> runs as root
CMD ["python", "app.py"]
Why it is unsafe: it pulls an unpinned, oversized base; it writes API_KEY into an ENV layer; it copies and then "removes" secret.pem (still recoverable from the earlier layer); and it runs as root.
Secure rewrite:
# secure.Dockerfile
# 1. Pinned, minimal, trusted base (replace <digest> with the real one)
FROM python:3.12-slim@sha256:<digest>
# 2. Non-root user
RUN useradd --system --no-create-home --shell /usr/sbin/nologin appuser
WORKDIR /app
# 3. Install deps in a cache-friendly, leak-free way
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# 4. Copy ONLY what the app needs, owned by the non-root user.
# A .dockerignore (below) keeps secrets out of the build context entirely.
COPY --chown=appuser:appuser . .
# 5. Drop to the unprivileged user before running
USER appuser
EXPOSE 8000
CMD ["python", "app.py"]
# .dockerignore -- keep secrets and junk OUT of the build context
.env
*.pem
*.key
.git
__pycache__/
Build and run with runtime hardening (lab commands):
# Build the secure image
docker build -f secure.Dockerfile -t myapp:1.0 .
# Run hardened: read-only FS, drop all Linux capabilities, no new privileges
docker run --rm \
--read-only \
--cap-drop=ALL \
--security-opt no-new-privileges \
--user appuser \
-p 8000:8000 \
myapp:1.0
Mitigation verification — test that the fix worked:
# (a) Confirm no secret is baked into the layers
docker history --no-trunc myapp:1.0 # should show NO API_KEY / secret values
# (b) Confirm the container does NOT run as root
docker run --rm myapp:1.0 id # expect uid != 0 (appuser), not uid=0(root)
# (c) Re-run the SAME checks against the insecure image to see the difference
docker build -f insecure.Dockerfile -t myapp:bad .
docker history --no-trunc myapp:bad # the API_KEY value is visible here
docker run --rm myapp:bad id # uid=0(root)
Expected outcome. For myapp:1.0, docker history shows the build steps but no secret values, and id reports a non-zero UID (appuser). For myapp:bad, the API_KEY value appears in the history output and id reports uid=0(root) — demonstrating exactly the two risks the secure version removes.
Edge cases. A read-only filesystem breaks apps that write to disk; mount a writable tmpfs for scratch space (--tmpfs /tmp) rather than dropping --read-only. useradd may differ on non-Debian bases (Alpine uses adduser). And RUN --mount=type=secret requires BuildKit (the default in modern Docker).
Walkthrough of the secure Dockerfile and the verification, in build order:
FROM python:3.12-slim@sha256:<digest> — Docker pulls exactly that pinned, minimal base. Because it is pinned by digest, the same bytes are used on every machine and every rebuild; because it is slim, far fewer packages (and CVEs) come along.RUN useradd ... appuser — creates a system user with no home directory and no login shell. This produces a layer; no app code is present yet. The user exists but is not yet active.WORKDIR /app — sets the working directory for following instructions (created if absent).COPY requirements.txt . then RUN pip install --no-cache-dir ... — dependencies are installed in their own layer before the app code. Because this layer only changes when requirements.txt changes, edits to your source code reuse the cached dependency layer, making rebuilds fast. --no-cache-dir avoids leaving pip's cache in the image.COPY --chown=appuser:appuser . . — copies the build context (already filtered by .dockerignore, so .env/*.pem never enter) and sets ownership to appuser, so the non-root process can read/write its own files.USER appuser — from this point the running process is unprivileged. This is the single most important hardening line.CMD ["python", "app.py"] — the default command, in exec form so the process is PID 1 and receives signals (clean shutdown).Now the verification trace:
| Step | Command | What it inspects | Pass result |
|---|---|---|---|
| a | docker history --no-trunc myapp:1.0 |
The build commands stored per layer | No secret strings appear |
| b | docker run --rm myapp:1.0 id |
The runtime UID of the process | uid is non-zero (appuser) |
| c | same checks on myapp:bad |
Contrast image | API_KEY visible; uid=0(root) |
The reason (b) returns a non-root UID is step 6 (USER appuser). The reason (a) shows no secret is that the secure file never uses ENV API_KEY=... and .dockerignore keeps secret files out of the context. The contrast in (c) makes the two failure modes concrete: a visible secret value and uid=0.
Mistake 1 — "Deleting" a secret in a later layer.
COPY id_rsa /root/.ssh/id_rsa # WRONG: secret is now in this layer permanently
RUN make build
RUN rm /root/.ssh/id_rsa # only hides it; earlier layer still has it
Why it is wrong: layers are immutable; rm adds a new layer that hides the file but cannot erase the earlier one. Fix: use a build secret — RUN --mount=type=secret,id=ssh_key make build — so the key is never written to any layer. Recognize it: docker history --no-trunc shows the COPY of a sensitive file.
Mistake 2 — Secrets in ENV or ARG.
ARG DB_PASSWORD # WRONG
ENV DB_PASSWORD=$DB_PASSWORD
Why it is wrong: both persist into the image and are readable via docker history / docker inspect. Fix: inject at runtime (docker run -e DB_PASSWORD=... from a secrets manager, or a mounted file). Recognize it: any credential-looking name in an ENV/ARG.
Mistake 3 — Using latest.
FROM node:latest # WRONG: unpinned, non-reproducible, moving target
Fix: FROM node:20.11-slim@sha256:<digest>. Recognize it: the absence of a version and digest; builds that "suddenly broke" with no code change.
Mistake 4 — Forgetting USER (running as root).
The Dockerfile installs and runs everything as root because no USER line is present. Fix: add a non-root user and USER appuser before CMD. Recognize it: docker run img id prints uid=0(root).
Mistake 5 — Copying the whole context (COPY . .) with no .dockerignore.
This sweeps .env, .git (with its history), and key files into the image. Fix: add a .dockerignore listing secrets and junk, and copy only what you need. Recognize it: image size larger than expected; sensitive files found when you extract layers.
Build-time errors
failed to compute cache key: ... not found — a COPY references a file excluded by .dockerignore or missing from the context. Check the path and your .dockerignore.--mount=type=secret fails or is ignored — BuildKit is not enabled. Use a modern Docker (BuildKit is default) or set DOCKER_BUILDKIT=1.Runtime errors
Permission denied after adding USER — the non-root user cannot write to a directory owned by root. Fix with COPY --chown=... or RUN chown -R appuser:appuser /app.--read-only — it needs to write somewhere. Add a writable --tmpfs /tmp (or a volume) instead of removing --read-only.Operation not permitted after --cap-drop=ALL — the app genuinely needs one capability; add it back narrowly with --cap-add=<NAME> rather than dropping the hardening.Logic / security issues
docker history --no-trunc <image> and, if needed, docker save <image> -o img.tar then extract and grep the layers in your lab. If the value appears, the secret is still baked in.docker run --rm <image> id. If uid=0, your USER line is missing or comes after the entrypoint.Questions to ask when it doesn't work
docker history --no-trunc)ARG, or ENV?Security & safety
This is a security topic, so the focus is on threat modeling and defense rather than C memory safety.
THREAT MODEL (container image)
Assets: app source, credentials/keys, the host kernel
Entry points: pulled base image, build context, the registry,
the running container's exposed ports
Trust boundaries: [build context] -> [image layers] -> [registry]
[container process] --|kernel|-- [host OS]
Key risks at each boundary:
* build context -> layers : secrets copied in (permanent)
* registry : untrusted/unpinned base = supply chain
* process -> host : root + breakout = host compromise
Defensive practices
COPY/ENV/ARG a credential. Use runtime injection or BuildKit build secrets. Add a .dockerignore for .env, *.pem, *.key, .git.USER; at runtime add --cap-drop=ALL, --security-opt no-new-privileges, and --read-only (with --tmpfs for scratch). Never mount the Docker socket into an untrusted container — it is equivalent to root on the host.API_KEY=<development-placeholder> in examples and docs.Authorization & ethics. Only inspect, build, and break out of images you own or are explicitly authorized to test, on local/isolated/lab systems (your machine, a container, or an intentionally vulnerable lab). Do not pull, probe, or attack images, registries, or hosts you do not control.
Concrete real-world use. Nearly every modern web service ships as a container image: a company builds an image in CI, scans it, pushes it to a private registry, and a Kubernetes cluster pulls and runs it across many nodes. A real and recurring incident class is secret leakage through public images — organizations have leaked cloud credentials by pushing images that contained baked-in .env files or keys, which scanners and attackers then harvested from public registries. The fixes in this lesson (no baked secrets, .dockerignore, scanning) directly prevent that.
Containers also power CI runners, database deployments, ML training jobs, and edge/IoT devices where a minimal, hardened image matters because resources are tight and updates are infrequent.
Professional best-practice habits
Beginner rules:
.dockerignore before your first COPY.ENV, ARG, or a COPYd file.USER.docker history on your own images and look for leaks.Advanced habits:
--read-only, --cap-drop=ALL, no-new-privileges, seccomp/AppArmor profiles) and enforce non-root via policy in the orchestrator.Beginner 1 — Spot the risks.
Objective: read a given Dockerfile and list every security issue. Requirements: identify at least the base-image, secret, and USER problems in the insecure example from this lesson, and state the fix for each in one line. Constraints: do not rewrite the file yet; just enumerate. Hint: read top to bottom and ask "does this end up in a layer?" Concepts: baked secrets, base images, non-root USER.
Beginner 2 — Add a .dockerignore and a non-root user.
Objective: harden a simple Python (or Node) image. Requirements: create a .dockerignore that excludes .env, *.pem, *.key, and .git; add a non-root user and a USER line before CMD. Example: building should still succeed and docker run <img> id should print a non-zero UID. Constraints: do not change the app code. Hint: use COPY --chown. Concepts: .dockerignore, non-root USER.
Intermediate 1 — Prove a secret leaked, then remove it.
Objective: demonstrate and fix a baked secret in your local lab. Requirements: build an image that includes a placeholder secret (API_KEY=<development-placeholder>), find it with docker history --no-trunc, then produce a fixed image where the same command shows no secret. Example I/O: before — the placeholder appears in history; after — it does not. Constraints: never use a real credential. Hint: compare moving the value to runtime -e. Concepts: layers, baked secrets, runtime injection.
Intermediate 2 — Pin and shrink the base image.
Objective: make a reproducible, smaller image. Requirements: replace a latest base with a specific minor version pinned by @sha256: digest, switch to a -slim (or distroless) variant, and confirm the image size dropped (docker images). Constraints: the app must still run. Hint: resolve the digest from the official image's registry page or docker inspect. Concepts: base images, pinning, minimal images.
Challenge — Multi-stage build with runtime hardening and verification.
Objective: produce a hardened image where build tools and any build-time secret never reach the final stage, then run it locked down. Requirements: (1) a multi-stage Dockerfile (a builder stage compiles/installs; a minimal final stage copies only the artifact); (2) use RUN --mount=type=secret for any build-time token so it is not in a layer; (3) final image runs as non-root; (4) run with --read-only --cap-drop=ALL --security-opt no-new-privileges plus a --tmpfs for scratch; (5) write the three verification commands (docker history, id, and a check that the app still works) and the expected results. Constraints: no real secrets; lab only. Hint: only the final stage's layers ship — use that to keep secrets and toolchains out. Concepts: multi-stage builds, build secrets, non-root USER, runtime hardening, mitigation verification.
COPY, ENV, or ARG a credential. Use runtime injection or BuildKit build secrets, and a .dockerignore.@sha256: digest), and scan them; avoid latest.USER and add runtime hardening (--read-only, --cap-drop=ALL, no-new-privileges).FROM <img>@sha256:..., COPY --chown=user:user, RUN --mount=type=secret, and USER appuser before CMD.docker history --no-trunc (no secrets) and docker run img id (non-zero UID). Most container findings are build-time mistakes — find them by reading the Dockerfile and running one or two inspection commands. Lab only, authorized systems only.