Compression and archives

Overview

Almost every file you download in bulk — a software release, a backup, a dataset, a set of photos — arrives as a single packaged file. Two completely different operations make that possible, and beginners constantly confuse them.

• Archiving takes many files and folders and bundles them into one file, preserving names, directory structure, and metadata (permissions, timestamps). The classic Unix tool is tar ("tape archive"). Archiving by itself does not make the data smaller.
• Compression takes a stream of bytes and makes it smaller by removing redundancy. Tools include gzip, bzip2, and xz. Compression by itself does not bundle multiple files.

Because you usually want both — one file and smaller — the two are combined. On Unix you tar first, then compress the result, giving .tar.gz (often written .tgz). On Windows and cross-platform tooling, the .zip format does both jobs inside one container.

This lesson builds directly on the previous one, Encoding. There you learned that bytes can be represented in different ways (Base64, hex, character sets) without changing the underlying information. Compression is the next step: instead of changing how bytes look, it changes how many bytes you need to store the same information. Both are reversible transformations of data — not security features.

Archives are not just a convenience format; they are also an attack surface. Any program that extracts an archive supplied by someone else is trusting the contents of that archive. Two well-known abuses — zip-slip (writing files outside the intended folder) and zip bombs (tiny input that explodes into a huge output) — are the security core of this lesson and connect forward to Logs and timestamps, where you will see how extraction events should be recorded.

Why it matters

Handling archives is everyday work. Backups arrive as .tar.gz. Software is shipped as .zip or .tar.xz. Datasets, log bundles, crash dumps, and forensic captures are all packaged. If you cannot confidently list and extract these, routine tasks stall.

The security stakes are concrete. Zip-slip and zip bombs are not theoretical — they are recurring vulnerability classes that have affected widely used libraries and applications across many languages. Any service that lets a user upload an archive (an avatar pack, a plugin, a document import, a CI artifact) and then extracts it can be attacked through the archive's contents rather than through the network protocol. Understanding the failure mode is the first step to writing extraction code that refuses malicious input instead of trusting it.

There is also an analyst angle: compressibility tells you something about data. If a file refuses to shrink, it is probably already compressed or encrypted. That single observation helps when triaging unknown blobs during incident response or reverse engineering.

Core concepts

1. Archiving vs. compression

Definition. Archiving concatenates multiple files (with their names and metadata) into one container file. Compression re-encodes a byte stream into fewer bytes by exploiting redundancy.

Why they are separate. On Unix the design philosophy is "one tool, one job." tar knows about files, directories, and permissions but nothing about shrinking data. gzip knows how to shrink a stream but nothing about file structure — it compresses exactly one stream. You pipe them together to get both.

  many files                 one bundle              one smaller file
  +-------+                  +----------+            +-------------+
  | a.txt |  --- tar --->    |  a.txt   | -- gzip -> | archive.tar |
  | b.txt |   (archive)      |  b.txt   |  (compress)|     .gz     |
  | dir/  |                  |  dir/... |            | (binary)    |
  +-------+                  +----------+            +-------------+

When NOT to combine the way you'd expect: .zip compresses each file separately and then stores them together. .tar.gz archives everything first, then compresses the whole bundle as one stream. That is why .tar.gz usually compresses better for many small similar files (cross-file redundancy is visible to gzip), while .zip lets you extract a single member without decompressing the rest.

Pitfall. Beginners say "I tarred the file to compress it." A plain .tar is the same total size as the inputs (plus a little header overhead). Bundling is not shrinking.

Knowledge check: You run tar -cf out.tar bigfolder/ and the result is slightly larger than the folder. Is that a bug? (Hint: think about per-file headers.)

2. The common formats

Pitfall. The extension is a hint, not a guarantee. A file named backup.zip could actually be a .tar.gz, or a renamed JPEG, or random data. The real format is determined by the magic bytes at the start of the file (e.g. gzip starts with 1f 8b, zip with PK\x03\x04). On Unix the file command reads these for you.

3. How lossless compression works (and the entropy tell)

Definition. Lossless compression reproduces the original bytes exactly on decompression. (Contrast with lossy compression like JPEG or MP3, which throws away detail you are unlikely to notice and cannot be reversed.)

How it works internally. Two ideas do most of the work:

• Dictionary/back-references (LZ family): repeated sequences are replaced by a short pointer that says "copy N bytes from M bytes ago."
• Entropy coding (Huffman, etc.): frequent symbols get short codes, rare symbols get long codes.

Input:  AAAAAAAA BBB AAAAAAAA
LZ:     A(x8) BBB <copy 8 bytes from 11 ago>   <- back-reference
        repeated runs and copies become tiny tokens

The entropy tell. "Entropy" here means how unpredictable the data is. Text and code are full of patterns, so they shrink a lot. Data that is already random-looking — encrypted blobs, already-compressed media like JPEG/MP3/MP4, or a .zip inside a .zip — has no leftover patterns, so it barely shrinks. Practical rule:

If a file refuses to compress, it is probably already compressed or encrypted.

When NOT to compress: do not gzip a JPEG, an MP4, or an encrypted file expecting gains — you may even add a few bytes of overhead.

Knowledge check: You compress two 10 MB files. report.txt becomes 1 MB; photo.jpg stays 10 MB. What does that tell you about each file's contents?

4. Zip-slip (path traversal in archives)

Definition. Zip-slip is a vulnerability where an archive entry's stored path contains directory-traversal segments (like ../) or an absolute path, so a naive extractor writes the file outside the intended extraction directory.

How it happens. An archive is just a list of (path, contents) pairs. The path is whatever the creator put there. A normal entry is docs/readme.txt. A malicious entry might be ../../../../etc/cron.d/evil or /home/user/.ssh/authorized_keys. If your code does open(extract_dir + "/" + entry_path) and blindly writes, the ../ parts walk back up and out.

intended:   /srv/uploads/job123/

entry path in archive:  ../../../../etc/cron.d/evil

naive join + write:
/srv/uploads/job123/../../../../etc/cron.d/evil
                 \__ resolves to __/  /etc/cron.d/evil   <- ESCAPED the sandbox

Defensive fix (the core idea): never trust the stored path. After joining, resolve to an absolute, canonical path and verify it still starts with the canonical extraction directory. Reject entries that escape, that are absolute, or (on systems that follow them) that are symlinks pointing outside. The related exercises path-traversal-check and detect-zip-slip drill exactly this check.

Pitfall. Checking the raw string for .. is not enough on its own — encodings, backslashes on Windows, and symlinks can all bypass a naive substring test. Canonicalize first, then compare.

5. Zip bombs (decompression denial of service)

Definition. A zip bomb is a small archive crafted so that decompressing it produces an enormous amount of data (or files), exhausting disk, memory, or CPU — a denial-of-service (DoS).

How it works. Compression ratios for highly repetitive data are huge: a few kilobytes of zeros can expand to gigabytes. Nested archives multiply the effect. The attacker spends almost nothing to send; the victim spends everything to expand.

Defensive fix. Decompress with limits: cap total output bytes, cap the number of entries, cap per-entry size, and refuse archives whose declared/observed expansion ratio is implausible. Stream and check as you go — stop the moment a limit is exceeded — rather than extracting first and measuring after.

Knowledge check (explain in your own words): Why is checking the compressed file size useless for stopping a zip bomb, and what should you measure instead?

Syntax notes

Common command-line patterns (Unix). tar flags read like sentences: c=create, x=extract, t=list, f=file, z=gzip, j=bzip2, J=xz, v=verbose.

# Create a gzip-compressed tar archive of a folder
tar -czf backup.tar.gz project/
#      | | |  \______ inputs
#      | | \________ output file name
#      | \__________ z = run gzip on the bundle
#      \____________ c = create archive

# LIST contents WITHOUT extracting (always do this first for untrusted archives)
tar -tzf backup.tar.gz

# Extract into a specific directory
mkdir -p out && tar -xzf backup.tar.gz -C out

# Identify the REAL format from magic bytes, ignoring the extension
file mystery.bin

# zip / unzip (cross-platform format)
zip -r site.zip site/
unzip -l site.zip      # -l lists without extracting
unzip site.zip -d out  # -d chooses the destination directory

Key habit: list before you extract (-t for tar, -l for unzip). Listing shows the stored paths so you can spot ../ or absolute paths before any file is written.

Lesson

Archiving bundles many files into one. Compression shrinks data by removing redundancy. The two are often combined.

Archive vs. compress

• tar bundles files into a single .tar file. By itself, it does not compress.
• gzip / bzip2 / xz compress a single stream, producing .gz, .bz2, or .xz.
• .tar.gz (also written .tgz) means tar first, then gzip. This is the Unix norm.
• zip does both archiving and compression in one container. This is the Windows and cross-platform norm.

How compression works (briefly)

Lossless compression replaces repeated patterns with shorter references.

Data that is already random or already compressed barely shrinks. Examples include encrypted blobs and JPEG images.

This gives a useful tell: high-entropy data that will not compress is often encrypted or already packed. (Entropy here means how random the data looks; truly random data has no patterns left to remove.)

Security relevance

• Loot often arrives as archives — backups (.tar.gz), exfiltration bundles, source dumps. Knowing how to list and extract them is routine work.
• Zip-slip. A malicious archive can contain paths like ../../etc/cron.d/x. If the extractor does not sanitise paths, the file is written outside the intended extraction directory. This is a real vulnerability class.
• Zip bombs. These are tiny archives that expand to enormous sizes — a denial-of-service trick.

Code examples

Below is a complete, safe listing-and-inspection session for an untrusted archive, followed by a short illustrative comparison. This is concept-track shell, not C. Every command here is read-only or writes only into an isolated out/ directory.

#!/usr/bin/env bash
# inspect-archive.sh — safely inspect an archive before trusting it.
# Usage: ./inspect-archive.sh suspicious.tar.gz
set -euo pipefail

archive="${1:?usage: inspect-archive.sh <archive>}"

# 1) Confirm the file exists.
if [ ! -f "$archive" ]; then
  echo "No such file: $archive" >&2
  exit 1
fi

# 2) Identify the REAL format from magic bytes (do not trust the extension).
echo "== Detected type =="
file "$archive"

# 3) Compare on-disk (compressed) size with the entropy idea.
echo "== Compressed size (bytes) =="
wc -c < "$archive"

# 4) LIST contents without extracting anything.
#    Look here for '../', leading '/', or absurd file counts (zip-slip / bomb tells).
echo "== Entries (no extraction yet) =="
case "$archive" in
  *.tar.gz|*.tgz) tar -tzf "$archive" ;;
  *.tar)          tar -tf  "$archive" ;;
  *.zip)          unzip -l "$archive" ;;
  *)              echo "Unknown extension; use 'file' output above." ;;
esac

# 5) Flag any suspicious paths automatically.
echo "== Suspicious-path scan =="
if tar -tzf "$archive" 2>/dev/null | grep -E '(^/|(^|/)\.\.(/|$))'; then
  echo "WARNING: entry escapes the extraction directory — do NOT extract blindly." >&2
else
  echo "No obviously-escaping paths found (still extract into an isolated dir)."
fi

What it does. It refuses to run without an argument, identifies the true format via file, prints the compressed size, lists every stored entry without writing a single file, and then scans those entries for absolute paths (^/) or traversal segments (..). The grep pattern matches a leading /, or a .. that is a whole path component.

Expected output (for a clean archive named project.tar.gz): a gzip compressed data line from file, a byte count, the list of members such as project/, project/main.c, project/README.md, and finally No obviously-escaping paths found ....

Expected output (for a malicious archive): the suspicious-path scan prints lines like ../../etc/cron.d/evil followed by the WARNING: message, and you stop.

Edge cases. The substring scan is a first-pass tripwire, not a complete defense — it does not catch symlink tricks or Windows backslash paths, and it only inspects .tar.gz in the grep step. Real extraction code must canonicalize each path at write time (see Mistakes). Also, set -euo pipefail makes the script abort on the first error and on use of an unset variable.

Line by line

Walking the inspection script top to bottom:

Values in flight: suppose archive="loot.tar.gz" and the archive contains app/config and ../../../../etc/cron.d/evil. The file step prints gzip compressed data; the list step prints both names; the grep step matches the second name because .. appears as a component, so the script prints the WARNING to stderr and you abort before any write occurs. The result is produced entirely from metadata — nothing is unpacked.

Common mistakes

Mistake 1 — "tar compresses my files."

Wrong: tar -cf big.tar hugefolder/ and expecting a smaller file. Why wrong: tar only bundles; the output is the same size plus headers. Corrected: add z (gzip), j (bzip2), or J (xz): tar -czf big.tar.gz hugefolder/. Recognize it: the .tar is not smaller than the inputs.

Mistake 2 — extracting an untrusted archive straight into the current directory.

Wrong: tar -xzf download.tar.gz in your home folder. Why wrong: archive entries may overwrite files, or escape via zip-slip. Corrected: always extract into a fresh, empty directory and list first:

tar -tzf download.tar.gz          # inspect names
mkdir -p sandbox && tar -xzf download.tar.gz -C sandbox

Recognize it: unexpected files appearing outside the folder you meant to fill.

Mistake 3 — defending against zip-slip with a substring check only.

Wrong (pseudocode): if entry.contains("..") reject; else write(extract_dir + "/" + entry). Why wrong: it misses absolute paths (/etc/...), Windows backslashes, encoded traversal, and symlinks that point outside. Corrected idea: canonicalize, then verify containment:

target   = canonical(join(extract_dir, entry))
basepath = canonical(extract_dir)
if not target.startswith(basepath + SEPARATOR):
    reject entry            # it escaped the sandbox
else:
    write(target)

Recognize it: a security review or a crafted test entry like ..%2f..%2fetc%2fpasswd slips through your filter.

Mistake 4 — extracting first, checking size after.

Wrong: unzip everything, then measure disk usage. Why wrong: a zip bomb has already filled the disk by then. Corrected: enforce a running cap on output bytes and entry count during extraction and abort the instant a limit is crossed.

Mistake 5 — trusting the file extension.

Wrong: deciding how to handle a file purely from .zip. Corrected: check magic bytes (file) and validate the real format before processing.

Debugging tips

"tar: Error is not recoverable: exiting now" or "gzip: stdin: not in gzip format." The compression flag does not match the real format. Run file archive to see what it actually is, then use the matching flag (-z gzip, -j bzip2, -J xz, or none for plain .tar).

unzip says "End-of-central-directory signature not found." The file is truncated, not really a zip, or was downloaded as text (corrupting bytes). Re-download in binary mode and confirm with file.

Extraction "succeeds" but files land in odd places. Suspect absolute or ../ paths in the entries. List the archive (tar -tzf / unzip -l) and inspect the names. Modern tar and unzip strip leading / and warn on .., but never rely on that — extract into an isolated directory.

Extraction hangs or fills the disk. Likely a zip bomb or an enormous legitimate archive. Cancel, then list entry count and uncompressed sizes (unzip -l shows uncompressed sizes; tar -tzvf is verbose) before retrying with limits and a dedicated volume.

Permission denied on extract. The archive stored paths or modes you cannot write. Extract as a normal user into a directory you own; do not extract untrusted archives as root.

Questions to ask when it won't work: What does file report? Did I list before extracting? Am I extracting into an empty, owned, isolated directory? Do any entry names start with / or contain ..? Is the uncompressed size plausible?

Memory safety

Security & safety

This is a non-security category lesson, but archive extraction is a classic place where untrusted input meets the filesystem, so treat it defensively.

Authorization & ethics. Only create, inspect, or detonate test archives (including any zip-bomb or zip-slip sample) on a machine you own or are explicitly authorized to use — ideally an isolated VM or container. Never deploy a deliberately malicious archive against systems you do not control. The goal here is prevention and detection, not attacking anyone.

Threat model (text diagram).

ASSETS:        files on the host filesystem, host availability (disk/CPU/RAM)
TRUST BOUNDARY: your extractor  |  the untrusted archive
ENTRY POINT:   user-uploaded / downloaded archive -> extract()
THREATS:       zip-slip  -> write outside extract dir (integrity / RCE)
               zip bomb  -> exhaust disk/memory      (availability / DoS)
               symlinks  -> redirect writes outside dir

Defensive practices.

• Canonicalize and contain: resolve each entry to an absolute path and confirm it stays inside the (canonical) extraction directory before writing. Reject absolute paths and .. components.
• Refuse dangerous entry types unless you specifically need them: symlinks, hardlinks, device nodes.
• Bound the output: cap total bytes, entry count, per-entry size, and compression ratio; stop on the first breach.
• Least privilege: extract as an unprivileged user, into a fresh directory, ideally inside a container with a disk quota.
• Detection / logging: log the archive's source, name, size, hash, entry count, and any rejected entries (and why rejected). This connects to the next lesson, Logs and timestamps. Never log secrets — if an archive contains credentials, log that one was present and quarantined, not its contents.

Verify your mitigations. Build a tiny test archive containing one normal entry and one ../escape entry on a throwaway VM, run your extractor, and confirm the escaping entry is rejected and no file appears outside the sandbox. Repeat with an oversized highly-repetitive entry to confirm your size cap aborts cleanly.

Real-world uses

Where this shows up.

• Software distribution: Linux source and binaries ship as .tar.gz / .tar.xz; Windows tools ship as .zip. Language ecosystems use archives under the hood (.jar is a zip; Python wheels are zips; npm packages are gzipped tarballs).
• Backups & DevOps: nightly backups, container image layers, and CI build artifacts are all archives.
• Web apps: "import these documents," "upload a theme/plugin," or "restore from backup" features extract user-supplied archives — exactly where zip-slip and zip bombs bite.
• Security & forensics: evidence and log bundles are archived; analysts use the entropy tell to spot encrypted or already-packed blobs.

Professional best-practice habits.

Beginner rules: always list before extract; extract into a fresh, isolated directory you own; verify the real type with file; never extract untrusted archives as root; keep a copy of the original archive until extraction is verified.

Advanced rules: enforce hard limits on output size/entry count/ratio; canonicalize and containment-check every path; reject symlinks/hardlinks/absolute paths by default; run extraction under least privilege with a disk quota or in a sandbox; record an audit log (source, hash, entry count, rejections) and verify file integrity with a checksum or signature before trusting the contents.

Practice tasks

Beginner 1 — Archive vs. compress (observe the difference). Create a folder with a few text files. Make a plain .tar, then a .tar.gz. Compare the three sizes (ls -l or wc -c). Requirements: report all three byte counts and a one-line explanation of why the .tar is not smaller than the inputs. Concepts: archiving vs. compression. Hint: tar -cf, then tar -czf.

Beginner 2 — The entropy tell. Compress two files: a large plain-text file and a JPEG (or any media file). Requirements: show the before/after sizes for each and explain in one sentence why the media file barely shrank. Output example: report.txt 5,000,000 -> 800,000 and photo.jpg 4,000,000 -> 3,995,000. Concepts: lossless compression, entropy. Hint: gzip -k file keeps the original.

Intermediate 1 — Safe inspection workflow. Write a short shell script that, given an archive, prints its real type (file), lists its entries without extracting, and reports the entry count. Requirements: must not write any file from the archive; must handle .tar.gz and .zip. Concepts: list-before-extract, magic bytes. Hint: tar -tzf, unzip -l, case on the extension.

Intermediate 2 — Zip-slip tripwire. Extend your script (or the lesson's) to scan listed entry names and reject any that are absolute (^/) or contain a .. path component, printing each offending entry. Requirements: exit non-zero if any suspicious entry is found. Input example: an archive containing ok/file and ../escape. Expected: it flags ../escape and exits non-zero. Concepts: path traversal, defensive validation. Hint: the grep -E pattern from the lesson is a starting point — note its limits.

Challenge — Safe extractor with containment and limits (lab only). Write an extractor (any language you know) that extracts into a fresh directory and, for each entry: canonicalizes the target path and refuses anything outside the extraction directory; rejects absolute paths and symlinks; and enforces a total-output-byte cap and an entry-count cap, aborting cleanly when exceeded. Requirements: on a throwaway VM, test it against (a) a normal archive, (b) one ../escape entry, and (c) one oversized repetitive entry; confirm (b) and (c) are rejected with no stray files written. Constraints: run as a non-root user; never test against systems you do not own. Concepts: canonicalization + containment, resource limits, least privilege, mitigation verification.

Summary

• Archiving bundles; compression shrinks. tar bundles many files into one; gzip/bzip2/xz compress a stream; .tar.gz does both (tar then gzip); .zip does both in one container (per-file).
• Key syntax: tar -czf out.tar.gz dir/ to create, tar -tzf / unzip -l to list before extracting, tar -xzf x.tar.gz -C out to extract into an isolated directory, and file to learn the real format from magic bytes.
• Entropy tell: data that refuses to compress is probably already compressed or encrypted.
• Two big risks: zip-slip (entry paths like ../../etc/... escape the extraction directory — defend by canonicalizing and verifying containment) and zip bombs (tiny input expands hugely — defend with size/entry/ratio limits enforced during extraction).
• Common mistakes: thinking tar compresses, extracting untrusted archives in place, defending zip-slip with a naive .. substring check, checking size only after extraction, and trusting the file extension.
• Remember: list first, extract into a fresh isolated directory as a non-root user, validate every path, bound the output, and log what you extracted — never the secrets inside it.

Practice with these exercises

• Zip Slip: Reject Unsafe Archive Entries
• Zip-Slip Safe Path Join
• Detect path-traversal patterns in a string