Compression — Make Bytes Smaller, Make Everything Faster

Compression is the art of finding patterns in data and replacing them with shorter codes. The original data can be recovered from those codes — that's lossless compression. Or you can discard tiny details humans won't miss and get even smaller files — that's lossy. The same idea has powered the internet since 1977, but new variants keep arriving because shaving 5% off a billion daily requests means millions of dollars saved or spent.

Picture the string "Hello World Hello World Hello World". It's 35 bytes written out in full. But notice — the phrase "Hello World" repeats three times. A compressor sees that pattern and writes it as [Hello World × 3] — just 15 bytes. It found the redundancy and replaced it with a compact reference. Decompression is just reading the reference and expanding it back. That's the entire idea. Everything else is refinement.

Here's a number that will change how you think about compression: AWS charges roughly $0.08 per GB of data transferred out to the internet. That sounds small. But at scale, it's the difference between a startup that stays profitable and one that bleeds cash.

Imagine your API returns typical JSON responses averaging 50 KB each. Your service gets 100 million requests per day. Do the math: 100M × 50 KB = 5,000 TB transferred — that's 5 petabytes of egress per day. At $0.08/GB, that's $400,000 per day, or $146 million per year. Enable gzip, and those 50 KB payloads compress down to roughly 12 KB. Same 100M requests: 1,200 TB per day. Bill: $96,000/day or $35M/year. That's a $111 million annual saving for one config line.

The CPU cost of compression (a few microseconds per request) is utterly trivial compared to those numbers. This is why engineers at companies like Cloudflare, Netflix, and Google treat compression as a first-class concern — not a nice-to-have.

The graph above uses real AWS pricing on a realistic workload. Your numbers will vary — payloads larger or smaller, traffic higher or lower, CDN pricing different — but the shape is always the same. Uncompressed dominates. Even gzip, a 32-year-old algorithm, cuts the bill by 75–80% on typical JSON. Brotli shaves another 10–15% on top of that.

Beyond cloud bills, compression also speeds up the user's experience. A 50 KB JSON response over a 4G mobile connection (typical ~10 Mbps downlink) takes ~40ms. At 12 KB, that drops to ~10ms. On a slow 3G connection (1 Mbps), 50 KB = ~400ms — a noticeable pause. 12 KB = ~96ms — feels instant. Compression is free latency improvement for your users.

Compression works because most real-world data is not random. Random data — like the output of a true random number generator — is already as compact as it can be. There's no pattern to exploit. But almost everything we actually store or transmit is deeply patterned.

Consider a photo of a clear blue sky. Millions of pixels, but they're almost all the same shade of blue. An image with a thousand pixels of exactly RGB(100, 149, 237) doesn't need to store that triple a thousand times — it can say "blue, 1000 times" and be vastly smaller. Or consider a text file: the word "the" appears roughly every 14 words in English. A compressor sees that repetition and replaces every occurrence after the first with a tiny back-reference. The more patterned your data, the more a compressor can save.

Different algorithms exploit different kinds of patterns. The four main pattern types — and the algorithm families that exploit them — are shown below.

In practice, modern compressors like zstandard and brotli layer these approaches: LZ77 handles the repeated-string redundancy first, then Huffman (or the faster ANS — Asymmetric Numeral Systems) handles the frequency skew in the remaining output. Combining multiple techniques is why modern compressors are so much better than RLE alone.

The three axes every compressor balances:

• Compression ratio — how small does the output get? Expressed as compressed_size ÷ original_size. Smaller is better. A ratio of 0.25 means "25% of original" — you saved 75%.
• Encode speed — how long does compressing take? This is paid once when writing or transmitting. For HTTP responses, this is paid on every request — so encode speed matters enormously.
• Decode speed — how long does decompressing take? This is paid every time someone reads the data. For web assets served to millions of browsers, decode speed is critical — you can't make users' CPUs faster.

Six terms come up in every compression discussion. You'll see them in documentation, interviews, and architecture reviews. Here they are in plain English first — then the precise definition.

Every compressor you use today traces back to a small number of seminal papers. Understanding the lineage demystifies why modern formats are the way they are. It's not arbitrary — each generation fixed a specific problem with the previous one.

Compression isn't free. It costs CPU on encode and decode. And it has one hard rule: you cannot compress already-compressed data. JPEGs, MP4 videos, and encrypted payloads are essentially random noise from the compressor's perspective — they have no patterns to exploit. Running gzip over a JPEG will make the file larger, not smaller, because the gzip header overhead gets added with zero savings. Knowing what to compress (and what not to) is as important as knowing how to compress.

The flowchart above gives you an instant answer for 90% of decisions. Let's add concrete numbers to the five main use cases:

Side-by-Side: Same JSON, Three Algorithms

Let's make the trade-offs concrete. The same JSON object — a realistic user profile — compressed with gzip, zstd, and brotli:

import gzip, json, time

payload = json.dumps({
    "user_id": "usr_8f3a2b1c",
    "name": "Alice Johnson",
    "email": "alice.johnson@example.com",
    "created_at": "2024-01-15T10:30:00Z",
    "plan": "pro",
    "preferences": {
        "theme": "dark",
        "language": "en",
        "notifications": True
    },
    "tags": ["power-user", "early-adopter", "api-access"]
}).encode()

t0 = time.perf_counter_ns()
compressed = gzip.compress(payload, compresslevel=6)  # default level
encode_ms = (time.perf_counter_ns() - t0) / 1e6

t0 = time.perf_counter_ns()
decompressed = gzip.decompress(compressed)
decode_ms = (time.perf_counter_ns() - t0) / 1e6

print(f"Original  : {len(payload):,} bytes")
print(f"gzip lvl6 : {len(compressed):,} bytes")
print(f"Ratio     : {len(compressed)/len(payload):.2f}  ({100*(1-len(compressed)/len(payload)):.0f}% savings)")
print(f"Encode    : {encode_ms:.3f} ms")
print(f"Decode    : {decode_ms:.3f} ms")

# Output on typical hardware:
# Original  : 258 bytes
# gzip lvl6 : 191 bytes  ← gzip header overhead is ~18 bytes; small payloads hurt
# Ratio     : 0.74  (26% savings)
# Encode    : 0.041 ms
# Decode    : 0.012 ms
#
# Note: compression ratio improves dramatically on larger payloads.
# At 50 KB, gzip typically achieves 75-85% savings on JSON.

import zstandard, json, time

payload = json.dumps({
    "user_id": "usr_8f3a2b1c",
    "name": "Alice Johnson",
    "email": "alice.johnson@example.com",
    "created_at": "2024-01-15T10:30:00Z",
    "plan": "pro",
    "preferences": {
        "theme": "dark",
        "language": "en",
        "notifications": True
    },
    "tags": ["power-user", "early-adopter", "api-access"]
}).encode()

cctx = zstandard.ZstdCompressor(level=3)  # level 3 = "everyday" — fast + good ratio
dctx = zstandard.ZstdDecompressor()

t0 = time.perf_counter_ns()
compressed = cctx.compress(payload)
encode_ms = (time.perf_counter_ns() - t0) / 1e6

t0 = time.perf_counter_ns()
decompressed = dctx.decompress(compressed)
decode_ms = (time.perf_counter_ns() - t0) / 1e6

print(f"Original  : {len(payload):,} bytes")
print(f"zstd lvl3 : {len(compressed):,} bytes")
print(f"Ratio     : {len(compressed)/len(payload):.2f}  ({100*(1-len(compressed)/len(payload)):.0f}% savings)")
print(f"Encode    : {encode_ms:.3f} ms")
print(f"Decode    : {decode_ms:.3f} ms")

# Output on typical hardware:
# Original  : 258 bytes
# zstd lvl3 : 178 bytes  ← better than gzip even at the fast default level
# Ratio     : 0.69  (31% savings)
# Encode    : 0.018 ms   ← ~2× faster encode than gzip
# Decode    : 0.008 ms   ← ~1.5× faster decode than gzip
#
# At level 19 (archive): further 5-10% better ratio, ~20× slower encode.
# Use level 3 for real-time; level 19 for backups.

import brotli, json, time

payload = json.dumps({
    "user_id": "usr_8f3a2b1c",
    "name": "Alice Johnson",
    "email": "alice.johnson@example.com",
    "created_at": "2024-01-15T10:30:00Z",
    "plan": "pro",
    "preferences": {
        "theme": "dark",
        "language": "en",
        "notifications": True
    },
    "tags": ["power-user", "early-adopter", "api-access"]
}).encode()

# Quality 4 = fast mode (good for dynamic responses)
# Quality 11 = maximum (only for static assets pre-compressed at deploy time)
t0 = time.perf_counter_ns()
compressed_q4  = brotli.compress(payload, quality=4)
encode_q4_ms = (time.perf_counter_ns() - t0) / 1e6

t0 = time.perf_counter_ns()
compressed_q11 = brotli.compress(payload, quality=11)
encode_q11_ms = (time.perf_counter_ns() - t0) / 1e6

print(f"Original      : {len(payload):,} bytes")
print(f"Brotli q=4    : {len(compressed_q4):,} bytes  ({encode_q4_ms:.3f} ms encode)")
print(f"Brotli q=11   : {len(compressed_q11):,} bytes  ({encode_q11_ms:.3f} ms encode)")

# Output on typical hardware:
# Original      : 258 bytes
# Brotli q=4    : 173 bytes  (0.022 ms encode) ← similar to zstd at small size
# Brotli q=11   : 155 bytes  (1.840 ms encode) ← ~40x slower but best ratio
#
# Brotli q=11 is used for CDN pre-compression:
# compress once at deploy, serve to millions of users.
# NEVER use q=11 for dynamic/real-time API responses —
# the encode cost (~2ms per response) would dominate request latency.

The code shows the key insight: compression ratio at small payload sizes (258 bytes) looks modest — 26–40% savings. But at realistic API payload sizes (5–50 KB), the same algorithms achieve 70–85% savings on JSON because there's far more repeated structure to exploit. Always benchmark with your actual payload sizes, not synthetic micro-examples.

Before we get to the clever algorithms, let's look at the dumbest one that actually works. Run-Length Encoding (RLE) has exactly one idea: if you see the same byte repeated several times in a row, write the count and the byte once instead of the byte many times. The string AAAAABBB (8 bytes) becomes 5A3B (4 bytes). That's it.

It's almost insultingly simple — which is exactly why it's a great place to start understanding compression. You can hold the whole algorithm in your head at once, which makes the WHY obvious: it only works when data contains long runs of identical values. Natural text almost never does (when did you last write "aaaaaaa" on purpose?), but bitmap images with large solid-colored regions absolutely do.

Where RLE Wins — and Where It Spectacularly Fails

RLE crushes data that has long runs of the same value. Classic real-world examples:

• Bitmap images with solid areas — a logo with a big white background has thousands of identical white pixels in a row. RLE can shrink those dramatically.
• Fax transmissions (CCITT Group 3/4) — faxed documents are mostly white (blank page) with occasional black lines. Runs of white pixels are very long.
• Classic Windows BMP files — Windows supported RLE-encoded BMP natively (BI_RLE8 and BI_RLE4 variants) for exactly this reason.

But try RLE on natural English text and it EXPANDS the file. Hello (5 bytes) becomes 1H 1e 2l 1o (8 bytes), because English text almost never has consecutive duplicate characters. RLE sees no runs to exploit, and its overhead makes things worse. This is the fundamental insight: the right algorithm depends entirely on the structure of your data.

# Naive RLE encoder: outputs (count, byte) pairs as a list
def rle_encode(data: bytes) -> list[tuple[int, int]]:
    if not data:
        return []
    result = []
    count = 1
    current = data[0]

    for byte in data[1:]:
        if byte == current and count < 255:  # cap run at 255 so count fits in 1 byte
            count += 1
        else:
            result.append((count, current))
            count = 1
            current = byte

    result.append((count, current))  # flush the last run
    return result

def rle_decode(encoded: list[tuple[int, int]]) -> bytes:
    return bytes(byte for count, byte in encoded for _ in range(count))

# Example:
data = b"WWWWWBBBWWW"
enc = rle_encode(data)
print(enc)  # [(5, 87), (3, 66), (3, 87)]  — 87='W', 66='B'
print(len(data), "bytes →", len(enc) * 2, "bytes")  # 11 bytes → 6 bytes

# Length-prefixed RLE: output is actual bytes [count, byte, count, byte, ...]
# count byte is packed so count=1 can be stored as literal (saves a byte for non-runs)
def rle_encode_bytes(data: bytes) -> bytes:
    if not data:
        return b""
    out = bytearray()
    i = 0
    while i < len(data):
        # count the run
        j = i + 1
        while j < len(data) and data[j] == data[i] and (j - i) < 127:
            j += 1
        run_len = j - i

        if run_len >= 3:
            # Encode as run: positive length byte (1–127) then the value
            out.append(run_len)
            out.append(data[i])
            i = j
        else:
            # Encode as literals: negative count byte (–1 to –128) then raw bytes
            # Use 0x80 flag to signal "literal block"
            # (simplified: just emit count+byte anyway for clarity)
            for k in range(run_len):
                out.append(1)
                out.append(data[i + k])
            i = j

    return bytes(out)

# Why length-prefix matters:
# PackBits (used in TIFF, PCX, old Mac formats) uses exactly this scheme.
# The sign bit distinguishes "N literal bytes follow" from "repeat next byte N times".
# This avoids the overhead of emitting count=1 for every non-run byte.
data = b"AAABCDDDDD"
print(rle_encode_bytes(data))

RLE only works on runs. What about text where every character appears once, but some characters appear much more often than others? That's where Huffman coding shines.

The idea, invented by David Huffman in 1952 as a student at MIT: instead of giving every character the same 8-bit code, give common characters short codes and rare characters long codes. In English text, 'e' appears ~13% of the time while 'z' appears ~0.07% of the time — so 'e' might get a 3-bit code and 'z' might get a 12-bit code. The average bits-per-character across the whole file comes out much less than 8.

The Algorithm in Plain English

1. Count how often each symbol appears. Scan the whole file once and build a frequency table.
2. Put each symbol in its own tiny one-node tree, weighted by frequency.
3. Repeatedly merge the two lowest-weight trees into one new tree whose weight is the sum. The two sub-trees become left (bit 0) and right (bit 1) children.
4. Stop when you have exactly one tree. The path from root to any leaf gives that symbol's code — left = 0, right = 1.

The result: a file that needs on average ~1.95 bits per symbol instead of 8. For 100 symbols, that's 195 bits instead of 800 — a 76% reduction. And you can get back the exact original file from the compressed bits plus the Huffman tree (which is stored in the file header).

The Genius: Prefix-Free Codes

There's a subtle brilliance here that makes Huffman codes actually usable. The decoder reads a stream of bits and needs to know when it's finished reading one code and starting the next. Huffman's tree construction guarantees that no short code is the beginning (prefix) of any longer code — this property is called being prefix-free.

Because Huffman codes are prefix-free, the decoder can read bits one at a time, follow the tree from root to leaf, emit the symbol when it hits a leaf, then immediately start at the root again — no look-ahead, no separator bytes needed.

Limitations and Real-World Use

Huffman coding requires knowing the frequencies upfront, which usually means a two-pass scan (one to count, one to compress). Modern variants like adaptive Huffman coding and arithmetic coding avoid the two-pass requirement. In practice, Huffman is never used standalone anymore — it's the entropy-coding layer INSIDE bigger algorithms: DEFLATE (gzip), JPEG, PNG, and MP3 all use Huffman internally. On English text alone, Huffman achieves roughly a 50-55% size reduction. Combined with a dictionary stage (LZ77), the modern algorithms you use every day are born.

Huffman coding squeezes common characters into fewer bits, but it can't help with longer repeated phrases. "the quick brown fox" appearing 50 times in a document still costs many bits per appearance, because Huffman codes individual characters, not sequences. That's the gap LZ77 fills.

Abraham Lempel and Jacob Ziv published this idea in 1977. The insight is elegant: the dictionary you use to find repeated sequences is just the data you've already encoded — the previous bytes of the file itself. No separate vocabulary file to ship. When you see text you've seen before, instead of writing it again, write a "back-reference": "go back N bytes and copy M bytes from there." The decoder re-reads from its own output — it holds the same bytes the encoder had, so it can reproduce any back-reference exactly.

The Sliding Window — How the Dictionary Scales

The "dictionary" LZ77 searches is just the N bytes immediately behind the current position — called the sliding window. When you move forward one byte, the window slides, dropping the oldest byte and gaining the newest. The window size is a fundamental trade-off knob:

• DEFLATE (gzip) — 32 KB window. Small enough to fit comfortably in 1990s hardware memory.
• Brotli — up to 16 MB window. Bigger window = more chance to find long matches in large web pages.
• Zstandard (level 22) — up to 128 MB window. At maximum level, can match text seen megabytes ago.

Bigger window = better ratio but more memory. On a 1990s PC with 4 MB RAM, a 128 MB window was impossible. On a 2024 server with 64 GB RAM, it's trivial — which is why modern compressors win so decisively over gzip.

Matches also have a minimum length — typically 3 bytes in DEFLATE. A 2-byte back-reference token itself costs 2 bytes to store, so matching fewer than 3 bytes doesn't save anything. A match of 3 bytes saves 1 byte; a match of 100 bytes saves ~98 bytes.

LZ77 alone achieves roughly a 70-80% reduction on English text. Combined with Huffman entropy coding, you get DEFLATE — the algorithm that has powered gzip, ZIP, and PNG for 30 years.

You've seen LZ77 and Huffman separately. Now meet their offspring: DEFLATE, the algorithm that combines them, and gzip, the file format that wraps DEFLATE with a header and checksum. These two are nearly synonymous in everyday conversation — "gzip a file" and "compress with DEFLATE" mean the same thing in practice.

Phil Katz designed DEFLATE in 1993 (for PKZIP), and Peter Deutch standardized it as RFC 1951. It's been 30 years, and gzip is still the default HTTP compression on roughly 70% of all web responses, according to the HTTP Archive. That's not because it's the best algorithm anymore — it's not — but because it runs everywhere without configuration. Understanding gzip is understanding the web's compression baseline.

The three DEFLATE block types matter because they let the compressor adapt: tiny payloads use fixed tables (no overhead to store a custom table), while large blocks build custom Huffman trees tuned to the exact frequency distribution of that block's content. The compressor chooses dynamically.

Why gzip Has Lasted 30 Years

Where gzip Shows Its Age

• Slow encode at high levels. gzip -9 is 4-5x slower than gzip -6 for a mere 2-3% ratio improvement. The diminishing returns are steep.
• Outdated ratio. Brotli and zstd consistently beat gzip-9's ratio while being faster. There's no compression scenario where gzip is the optimal choice anymore — it wins only on compatibility.
• No dictionary training. gzip can't be trained on your specific data shape. A JSON API that always returns the same keys gets no special benefit; zstd dictionary training would help enormously.

Imagine you could tell a compressor: "I know you're going to compress a web page. Before you even start, I'll give you a dictionary of 13,000 common web phrases — <html>, </div>, function, Content-Type: application/json, charset=utf-8 — and you can match against those as if they were already in your sliding window." That's Brotli.

Google released Brotli in 2013, designed specifically for HTTP compression. Every other compressor we've looked at starts from nothing — the sliding window is empty at byte zero. Brotli starts with a 13,000-entry static dictionary of common HTML/CSS/JS phrases baked into the algorithm itself. Even a 200-byte JSON response can match against that dictionary. That's why Brotli wins so decisively on small web payloads where gzip can barely compress at all.

Why Brotli Wins on the Web

• 10-15% smaller than gzip on HTML/CSS/JS at comparable encode levels. For a 100 KB CSS bundle, that's 10-15 KB saved on every page load, every user, every day.
• Small payloads compress dramatically better. A 500-byte API response compressed with gzip is barely smaller than the original — the sliding window has no history. Brotli's static dictionary has 13,000 matching phrases loaded from the start. That 500-byte JSON payload gets real compression.
• Comparable decode speed to gzip. Decompression is always fast — it's encoding that's expensive. Brotli level 4-6 encodes at a similar speed to gzip level 9 with better output.
• Universal browser support since 2017. Chrome, Firefox, Safari, and Edge all support Accept-Encoding: br. HTTPS is required (Brotli is only served over secure connections by browser spec).

The One Place Brotli Loses

Brotli level 11 (maximum) is significantly slower to encode than gzip. It's too slow for on-the-fly dynamic compression of large responses. The solution: use level 11 for pre-compressing static assets (CSS, JS, images) at build time, and level 5-6 for dynamic responses in flight. Most CDNs and frameworks do exactly this automatically.

const { brotliCompress, brotliDecompress, constants } = require('zlib');
const { promisify } = require('util');

const brotliCompressAsync = promisify(brotliCompress);
const brotliDecompressAsync = promisify(brotliDecompress);

async function compressWithBrotli(data, quality = 6) {
  // quality 0-11: 0=fastest/worst, 11=slowest/best
  // For dynamic HTTP responses: quality 4-6 (fast, good ratio)
  // For static asset pre-compression: quality 11 (slow, best ratio)
  const compressed = await brotliCompressAsync(data, {
    params: {
      [constants.BROTLI_PARAM_QUALITY]: quality,
      // BROTLI_PARAM_MODE: TEXT (0), GENERIC (2), FONT (3)
      [constants.BROTLI_PARAM_MODE]: constants.BROTLI_MODE_TEXT,
    }
  });
  return compressed;
}

async function main() {
  const html = Buffer.from('<!DOCTYPE html><html><body><h1>Hello World</h1></body></html>'.repeat(100));
  console.log('Original:', html.length, 'bytes');

  const br6 = await compressWithBrotli(html, 6);
  console.log('Brotli-6:', br6.length, 'bytes', `(${((1 - br6.length/html.length)*100).toFixed(1)}% savings)`);

  const br11 = await compressWithBrotli(html, 11);
  console.log('Brotli-11:', br11.length, 'bytes', `(${((1 - br11.length/html.length)*100).toFixed(1)}% savings)`);

  // Roundtrip check
  const decompressed = await brotliDecompressAsync(br6);
  console.assert(decompressed.equals(html), 'Decompression mismatch!');
  console.log('Roundtrip OK');
}

main();

# nginx.conf — serve pre-compressed .br files, fallback to gzip
# Requires: ngx_brotli module (compile-time) OR nginx 1.25.1+ with --with-brotli

http {
    # ── Brotli dynamic compression ──────────────────────
    brotli on;
    brotli_comp_level 6;          # For dynamic: 4-6 is the sweet spot
    brotli_types text/html text/css application/javascript
                 application/json image/svg+xml;

    # ── gzip fallback (browsers without br support) ─────
    gzip on;
    gzip_comp_level 6;
    gzip_types text/html text/css application/javascript application/json;
    gzip_vary on;                 # Adds "Vary: Accept-Encoding" header

    server {
        listen 443 ssl;

        # ── Static assets: serve pre-compressed files ───
        # At build time: brotli -q 11 app.js → app.js.br
        # At build time: gzip -9 app.js → app.js.gz
        location /static/ {
            brotli_static on;     # Serve .br files automatically if client supports it
            gzip_static on;       # Serve .gz files as fallback
        }

        # ── API: dynamic compression ─────────────────────
        location /api/ {
            proxy_pass http://backend;
            # Brotli + gzip already enabled globally above
        }
    }
}

# Comparing Brotli quality levels 1, 6, 11 on a 1 MB JS bundle
# Run from the command line after installing: brew install brotli

INPUT="bundle.js"    # 1,024 KB uncompressed JavaScript

echo "=== Brotli Level Comparison ==="
for LEVEL in 1 4 6 9 11; do
  OUTPUT="bundle.br.level${LEVEL}"
  TIME_START=$(date +%s%3N)
  brotli --quality=${LEVEL} --output=${OUTPUT} ${INPUT}
  TIME_END=$(date +%s%3N)
  SIZE=$(wc -c < ${OUTPUT})
  ELAPSED=$((TIME_END - TIME_START))
  RATIO=$(echo "scale=1; (1 - ${SIZE}/1048576) * 100" | bc)
  echo "Level ${LEVEL}: ${SIZE} bytes  (${RATIO}% smaller)  encode: ${ELAPSED}ms"
done

# Expected output (approximate for a typical minified JS file):
# Level  1:  312 KB  (69.5% smaller)  encode:  8ms   ← dynamic responses
# Level  4:  285 KB  (72.1% smaller)  encode: 18ms
# Level  6:  271 KB  (73.5% smaller)  encode: 35ms   ← good default
# Level  9:  248 KB  (75.8% smaller)  encode: 210ms
# Level 11:  238 KB  (76.8% smaller)  encode: 1240ms ← pre-compress static assets only

# Compare to gzip-9: ~258 KB (74.7% smaller), encode ~90ms
# Brotli-11 wins ratio; Brotli-6 wins speed + decent ratio

If you could design a compressor today, knowing everything about modern hardware, what would you build? Facebook's Yann Collet answered that question in 2016 with Zstandard (zstd). The goal was simple and audacious: beat gzip's compression ratio AND beat gzip's speed, simultaneously, without compromise. Four years of iteration later, he did it.

zstd doesn't win by inventing a new fundamental algorithm — it's still LZ77 + entropy coding at its core. It wins through engineering excellence: a larger sliding window, a better entropy coder (Finite State Entropy, an ANS variant), 22 granular compression levels, built-in multi-thread support, and the killer feature gzip never had: dictionary training. You can feed zstd a sample of your actual data, and it builds a custom shared dictionary that makes tiny messages compress dramatically better.

The chart tells the story: every zstd level sits above or to the right of gzip on the ratio/speed curve. At level 3 (the default), zstd encodes at roughly 2.5x gzip's speed while matching gzip's ratio. At level 19, it beats gzip-9 ratio by ~8-10% with comparable encode time. There is genuinely no scenario where gzip is the better technical choice versus zstd — only compatibility requirements keep gzip alive.

zstd's Four Superpowers

Dictionary Training Visualized

Where zstd is Winning

• Linux kernel — default compression for kernel modules since Linux 5.9 (2020). Previously gzip; zstd boots faster and produces smaller modules.
• Apache Iceberg / Parquet / ORC — the best columnar compression for analytics workloads. Replaces Snappy as the default in many configurations.
• RocksDB / MongoDB / Cassandra — replacing Snappy and gzip for SSTable compression in storage engines.
• Facebook internal RPC — replaced gzip for essentially all internal service communication, saving significant compute and bandwidth at scale.
• GitHub — uses zstd for pack file compression in git.

# Install: brew install zstd  OR  apt install zstd

# Basic usage
zstd file.log                 # Compress → file.log.zst (default level 3)
zstd -d file.log.zst          # Decompress → file.log
zstd -19 file.log             # Level 19 (high ratio, slow encode)
zstd -1  file.log             # Level 1  (fast, moderate ratio)

# Multi-threaded (uses all CPU cores)
zstd --threads=0 big-dataset.json  # 0 = auto-detect CPU count

# Streaming (pipe-friendly)
tar --use-compress-program=zstd -cf archive.tar.zst ./data/
tar --use-compress-program=zstd -xf archive.tar.zst

# Compare with gzip on the same file
zstd -3 -k file.log -o file.log.zst3  # -k keeps original
gzip -6 -k file.log

ls -lh file.log file.log.zst3 file.log.gz
# Typical output for a 100MB server log:
# file.log        100 MB  (original)
# file.log.zst3    19 MB  (81% smaller, encodes in 0.3s)
# file.log.gz      22 MB  (78% smaller, encodes in 0.9s)

# Step 1: Collect sample messages (separate files or split a log)
mkdir samples/
# Assuming you have many small JSON files (or split one large file):
split -l 1 messages.ndjson samples/msg_  # one JSON line per file

# Step 2: Train the dictionary from samples
zstd --train samples/* -o my_api.dict
# Output: Completed with 1024 files, dictionary saved to my_api.dict
# my_api.dict is typically 32-112 KB

# Step 3: Compress using the dictionary
zstd -D my_api.dict message.json -o message.json.zst
zstd -D my_api.dict -d message.json.zst  # decompress (needs same dict!)

# Step 4: Compare compression with vs without dict
echo "Without dict:"
zstd -3 -k message.json
ls -lh message.json message.json.zst

echo "With dict:"
zstd -D my_api.dict message.json -o message_dict.json.zst
ls -lh message.json message_dict.json.zst

# Typical result for a 500-byte JSON message:
# Without dict: 500 bytes → 320 bytes  (36% savings)
# With dict:    500 bytes → 80 bytes   (84% savings)
# Dict overhead: dict ID is stored in frame header (~4 bytes)

import zstandard as zstd  # pip install zstandard
import time, os

# Load a 1 MB JSON file for testing
with open("data.json", "rb") as f:
    data = f.read()

original_size = len(data)
print(f"Original: {original_size:,} bytes ({original_size/1024:.1f} KB)")
print(f"{'Level':<6} {'Size':<10} {'Savings':<10} {'Encode ms':<12} {'MB/s':<8}")
print("-" * 50)

for level in [1, 3, 6, 9, 12, 19, 22]:
    compressor = zstd.ZstdCompressor(level=level)

    # Time encode
    start = time.perf_counter()
    compressed = compressor.compress(data)
    elapsed_ms = (time.perf_counter() - start) * 1000

    size = len(compressed)
    savings = (1 - size / original_size) * 100
    speed_mb = (original_size / 1024 / 1024) / (elapsed_ms / 1000)

    print(f"{level:<6} {size:<10,} {savings:<10.1f}% {elapsed_ms:<12.1f} {speed_mb:<8.1f}")

    # Verify roundtrip
    decompressor = zstd.ZstdDecompressor()
    assert decompressor.decompress(compressed) == data, "Roundtrip failed!"

# Expected output (approximate, 1 MB minified JSON):
# Level  Size       Savings    Encode ms    MB/s
# 1      310,000    70.0%      1.8          556
# 3      287,000    72.7%      3.2          312
# 6      274,000    73.9%      7.1          141
# 9      265,000    74.8%      18.0          55
# 12     260,000    75.2%      55.0          18
# 19     248,000    76.4%      180.0          5.5
# 22     243,000    76.8%      620.0          1.6

Most compression tools try to shrink your data as much as possible. But sometimes that's the wrong goal. Imagine you're a database writing 500 MB of hot data to disk every second. If compression takes 10 ms, you've just bottlenecked every write. What you actually want is compression so fast it's nearly invisible — where the CPU cost is smaller than the saved I/O time. That's the idea behind Snappy and LZ4.

Both were released in 2011 — Snappy from Google, LZ4 from Yann Collet (who would later build zstd). Both compress at roughly 500 MB/sec, which is close to the speed of a modern memory bus. They don't try for the smallest output; they aim for something that barely slows you down. The trade-off is about 30% worse compression ratio than gzip. On hot, frequently-read data that trade-off almost always wins.

Where Speed-First Compression Wins

These are the three scenarios where "fast enough" beats "smallest possible":

• Network-attached storage — when a server is streaming data between RAM and a network disk, compression is on the critical path. If the compressor can't keep up with the memory bus (~50 GB/s modern servers), you slow everything down. Snappy and LZ4 stay well below that ceiling.
• Real-time log pipelines — tools like Logback and Kafka producers compress message batches before sending. At 100k events/second, even a 1 ms compression overhead per batch compounds to seconds of lag. LZ4 compresses a 1 MB log batch in under 2 ms.
• Database storage layers — RocksDB, Cassandra, and MongoDB all ship with Snappy as the default compressor for active ("hot") data. The reasoning is explicit: hot data is read and written constantly; shaving 30% off ratio is not worth doubling the CPU cost on every read path.

Text compressors can't throw anything away — every byte of code or JSON must survive exactly. Images are different. A human looking at a photograph simply cannot see tiny differences in high-frequency detail — the flickering of light and shadow at the boundary of a leaf against sky. JPEG (finalized in 1992) was the first widely-deployed format to exploit this. It discards the detail you can't see, and in doing so achieves 10–20× compression on photographs with no perceptible quality loss.

The result was transformative. Before JPEG, a single photo over dial-up took minutes. After JPEG, the visual web became possible. Understanding how it works teaches you something deeper: lossy compression is about exploiting the limits of the receiver — in this case, human vision.

The 5 Steps, Plain English

1. Color space transform (RGB → YCbCr). RGB stores red, green, and blue light levels. JPEG converts to YCbCr: Y is brightness (luminance), Cb is "how blue-ish," Cr is "how red-ish." Why? Because human eyes are ~10× more sensitive to brightness differences than color differences. By separating them, JPEG can treat each channel differently.
2. Chroma subsampling. The Cb and Cr color channels are averaged — every 4 neighboring pixels share one color sample (this is called 4:2:0 subsampling). The brightness channel keeps full resolution. This step alone halves the color data, and on natural photographs humans almost never notice because our color vision is lower-resolution than our brightness vision.
3. DCT — the frequency transform. Each 8×8 block of pixels is run through a Discrete Cosine Transform (DCT). DCT converts 64 pixel values into 64 frequency components. The top-left component (DC) is the average color of the block. Components toward the bottom-right represent increasingly fine, high-frequency detail — sharp edges, fine texture. The image is now in "frequency space" instead of "pixel space."
4. Quantization — where the loss actually happens. Each frequency component is divided by a quality-dependent integer and rounded. High-frequency components get divided by large numbers — which rounds them to zero. Low-frequency components (the coarse shapes) survive. The "quality slider" in Photoshop or ImageMagick directly controls these divisor values. Smaller divisors = more data survives = larger file = better quality.
5. Entropy coding. After quantization, most high-frequency coefficients are zero. JPEG runs a zigzag scan (filling the 8×8 block diagonally, naturally grouping zeros together), then Run-Length Encodes the runs of zeros, then Huffman codes the whole stream. This is the lossless final step — it squeezes the already-quantized data without further quality loss.

Real numbers: JPEG quality 85 (the web standard) typically achieves 10–15× compression on natural photographs with no perceptible loss. Quality 50 reaches 25–30× but introduces visible blocking on edges. JPEG was the dominant image format on the web for nearly 30 years (1995–2022). It's only now being replaced by WebP and AVIF in most new deployments.

Thirty years after JPEG, the web finally has better options. Three formats have emerged to replace it — each with different engineering trade-offs. None has fully "won" yet, because the web moves at the speed of browser adoption, not algorithm improvement. But if you're building anything image-heavy today, you need to know these.

The Pragmatic Multi-Format Serve Pattern

Because no single format has 100% browser support, the correct approach is to offer multiple formats and let the browser pick. HTML's <picture> element was built for exactly this:

<!-- Browser tries formats in order — uses first one it supports -->
<picture>
  <!-- Best: AVIF if browser supports it (~90% of users, 2024) -->
  <source type="image/avif" srcset="photo.avif">

  <!-- Fallback: WebP (~95% of users) -->
  <source type="image/webp" srcset="photo.webp">

  <!-- Final fallback: JPEG (universal) -->
  <img src="photo.jpg" alt="Product photo" width="800" height="600">
</picture>

<!-- Result: AVIF users get 50% smaller file.
            WebP users get 30% smaller. JPEG users get baseline.
            Zero JS required — pure HTML negotiation. -->

# Cloudflare / Fastly / CloudFront all do this automatically.
# If you self-host via nginx + libvips or imageproxy:

location ~* \.(jpg|jpeg|png)$ {
    # Check if browser accepts AVIF
    if ($http_accept ~* "image/avif") {
        rewrite ^(.+)\.(jpg|jpeg|png)$ $1.avif break;
    }
    # Otherwise try WebP
    if ($http_accept ~* "image/webp") {
        rewrite ^(.+)\.(jpg|jpeg|png)$ $1.webp break;
    }
    # Fallback to original
}

# Pre-generate AVIF and WebP variants at deploy time:
# convert input.jpg -quality 80 output.avif
# cwebp -q 80 input.jpg -o output.webp

Real numbers: Switching from JPEG to AVIF typically saves 40-55% on image bandwidth. At the scale of a site with 10M daily image views averaging 200 KB per image, that's 400-550 TB/month saved. Cloudflare Images and AWS CloudFront both perform automatic format conversion — you store one JPEG; they serve the best format for each browser.

Video is, by far, the dominant consumer of internet bandwidth — roughly 80% of all traffic is video. The scale is almost incomprehensible: without compression, a single 1080p hour of uncompressed video at 24fps would be about 750 GB. With modern compression (H.264), that same hour is 2-4 GB. That's a 200× reduction. At Netflix's scale, this difference translates to billions of dollars per year in bandwidth costs.

Video compression builds on image compression (each frame is basically a JPEG) but adds a second axis of redundancy that's unique to video: time. Most video frames are nearly identical to the frame before. A talking head video has a static background. A panning shot moves the whole frame but changes nothing about the objects in it. Exploiting temporal redundancy is where the real savings come from.

The 3 Big Tricks

The 4 Dominant Video Codecs

Real numbers: Netflix encodes every title in 5+ codecs (H.264, H.265, VP9, AV1) and 20+ bitrate/resolution combinations — up to 120 variants per title. AV1 saves ~30% bandwidth vs H.264 across their catalog. At Netflix's ~15% of global internet traffic, that ~30% saving is worth billions of dollars per year in egress fees.

Every production storage system you'll encounter offers compression. But the knobs, defaults, and trade-offs are all different — and enabling compression on the wrong workload can spike CPU from 30% to 80% and double your tail latency. This section gives you the practical map: what each system does, which compressor it uses by default, and what levers you have.

The core trade-off is always the same: CPU time in exchange for I/O time. Compression wins when your bottleneck is disk or network bandwidth. It loses when your bottleneck is CPU. Most modern cloud workloads are I/O-bound, so compression usually pays off — but always measure with your actual workload before enabling it on a live system.

Real numbers: Discord's famous switch from Cassandra LZ4 to zstd-9 saved multiple petabytes of storage. Most systems see 40-70% storage reduction with modern compressors on typical application data. Column-oriented formats like Parquet often see 80-90% reduction because identical values in a column compress near-perfectly.

Compression isn't only about what you store — it's also about what you send over the wire. HTTP, gRPC, Kafka, and S3 all support compression in some form. The good news: it's usually just one header or one config option. The operational details, however, matter a lot — especially when your traffic passes through CDNs, proxies, and multiple application layers.

Code: Three Transport Configurations

http {
    # Enable gzip (broad compatibility)
    gzip on;
    gzip_vary on;           # Vary: Accept-Encoding header for CDN caching
    gzip_proxied any;       # Compress even for proxied requests
    gzip_comp_level 6;      # Sweet spot: good ratio, reasonable CPU
    gzip_min_length 1000;   # Don't compress tiny responses (overhead not worth it)
    gzip_types
        text/plain text/css text/javascript
        application/javascript application/json
        application/xml image/svg+xml;

    # Brotli (better than gzip, ~95% browser support)
    # Requires ngx_brotli module: https://github.com/google/ngx_brotli
    brotli on;
    brotli_comp_level 6;    # 1-11; level 6 = good ratio at reasonable speed
    brotli_types
        text/plain text/css application/javascript
        application/json image/svg+xml;

    # zstd (newest — requires ngx_zstd module)
    # zstd on;
    # zstd_level 3;
}

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

public class CompressedProducer {
    public static KafkaProducer<String, String> create() {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // zstd is the modern recommendation (Kafka 2.1+)
        // Options: "none", "gzip", "snappy", "lz4", "zstd"
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");

        // Larger batches = more redundancy = better compression
        // Default: 16384 (16 KB). Increase for compression workloads.
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 131072);  // 128 KB

        // Wait up to 10ms to fill a batch before sending
        // Gives more messages to compress together
        props.put(ProducerConfig.LINGER_MS_CONFIG, 10);

        return new KafkaProducer<>(props);
    }

    // WHY batch-level compression wins:
    // 1000 log messages compressed together: ~20:1 ratio (logs are repetitive JSON)
    // Each message compressed individually: ~3:1 ratio
    // The batch context provides the dictionary that makes compression shine.
}

package main

import (
    "context"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
    "google.golang.org/grpc/encoding/gzip" // registers gzip codec
)

func main() {
    // Method 1: default compression for ALL RPCs on this channel
    conn, err := grpc.Dial(
        "server:50051",
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithDefaultCallOptions(grpc.UseCompressor(gzip.Name)),
    )
    if err != nil {
        panic(err)
    }
    defer conn.Close()

    client := pb.NewMyServiceClient(conn)

    // Method 2: per-RPC compression (override per call)
    resp, err := client.GetLargeData(
        context.Background(),
        &pb.Request{Id: 123},
        grpc.UseCompressor(gzip.Name), // only THIS call is compressed
    )

    // WHY gzip and not something faster?
    // gRPC's built-in compressor is gzip-only by default.
    // For zstd or snappy, implement grpc/encoding.Compressor interface:
    //
    //   type Compressor interface {
    //     Compress(w io.Writer) (io.WriteCloser, error)
    //     Decompress(r io.Reader) (io.Reader, error)
    //     Name() string
    //   }
    //
    // Register with: encoding.RegisterCompressor(myCompressor)
}

Real numbers: Cloudflare reports brotli at the edge cuts global egress 35-55% for text assets (HTML, CSS, JS) compared to no compression. Kafka with zstd versus no compression typically delivers 4-8× throughput improvement on realistic application payloads. For a service processing 1 GB/s of Kafka messages, that's the difference between 8 brokers and 1.

Reading benchmarks online is helpful, but the only numbers that matter are the ones you get from your actual data. Log lines are very different from JSON API responses, which are very different from sensor readings. The tools below let you measure what happens when your specific bytes go through a compressor — not someone else's test file.

# clone and build (one-time)
git clone https://github.com/inikep/lzbench.git
cd lzbench && make

# benchmark against your data sample
./lzbench -ebrotli/6,gzip/6,zstd/3,lz4,snappy ./data-sample.json

# example output columns:
# Compressor         Ratio    Encode MB/s    Decode MB/s
# brotli 1.0.9 -6   3.21     22             342
# gzip 1.10 -6      2.89     61             310
# zstd 1.5 -3       3.04     435            1200
# lz4 1.9            2.11     650            4100
# snappy 1.1        1.97     460            2000
#
# zstd wins on ratio+speed for text JSON.
# lz4/snappy win on speed if ratio is secondary (log pipelines).
# brotli wins on ratio if encode speed is not a constraint.

# Step 1: gather representative samples (min 100 recommended)
mkdir samples
cp prod-api-payloads/2024-01-*.json samples/

# Step 2: train — output is a 112 KB binary file
zstd --train samples/*.json -o api-dict.zst

# Step 3: compress with dictionary
zstd -D api-dict.zst -3 payload.json -o payload.zst

# Step 4: decompress (must supply the same dict on the other end)
zstd -d -D api-dict.zst payload.zst -o payload_out.json

# Ratio comparison on a 400-byte JSON message:
#   Without dict:  400 → 310 bytes  (1.29:1)
#   With dict:     400 →  42 bytes  (9.5:1)
# WHY: the dict pre-loads field names like "timestamp", "userId",
# "eventType" that appear in every message — the compressor
# back-references them instead of re-encoding them each time.

# nginx config snippet (requires ngx_brotli module)
gzip            on;
gzip_types      text/plain text/css application/json application/javascript;
gzip_min_length 256;
gzip_comp_level 6;

brotli          on;
brotli_types    text/plain text/css application/json application/javascript;
brotli_comp_level 6;

# Test brotli negotiation — curl sends Accept-Encoding: br
curl -H "Accept-Encoding: br" -I https://yoursite.com/api/data
# Expected response headers:
#   Content-Encoding: br
#   Vary: Accept-Encoding

# Test gzip fallback
curl -H "Accept-Encoding: gzip" -I https://yoursite.com/api/data
# Expected:
#   Content-Encoding: gzip

# Measure actual compressed body size
curl -H "Accept-Encoding: br" -s https://yoursite.com/api/data | wc -c
# Compare to uncompressed:
curl -H "Accept-Encoding: identity" -s https://yoursite.com/api/data | wc -c

Compression is one of those topics where a little knowledge leads to confidently wrong decisions. These six misconceptions show up constantly in code reviews, architecture discussions, and incident post-mortems. Each one sounds reasonable until you look at the actual numbers.

Five incidents where compression went wrong in ways engineers didn't see coming. Each one changed how the industry thinks about a specific aspect of compression safety.

Everything from the page distilled into eight actionable rules. These are the decisions you should make by default, with the reasoning that justifies each one — so you can override them intelligently when your situation differs.

The questions that come up every time someone first works with compression in a production system. Answers are concrete and skip the "it depends" whenever a clear default exists.

Compression doesn't exist in isolation — it's always compressing something (a serialization format, an HTTP response, a database page) carried somewhere (a protocol, a CDN, a storage engine). Each of these topics connects directly to what you've studied here.