Redis — In-Memory Data Structures at Microsecond Speed

Redis lives in RAM — that is the whole insight. Everything else (speed, rich data structures, sub-millisecond latency) is a direct consequence of that one choice.

Let's start with a story you will live through at some point in your career. It does not require any prior knowledge — just follow the numbers.

The situation: your top-products page

You built an e-commerce site. The homepage shows the top 20 best-selling products. To generate that list, your application runs five SQL queries: fetch products, fetch inventory counts, fetch ratings, fetch review counts, fetch discount info. Together they take about 80 milliseconds — fast enough in development where you are the only user.

Then your site gets popular. Traffic climbs to 10,000 requests per second during a sale. Let's do the math:

• 10,000 req/sec × 5 queries = 50,000 SQL queries per second hitting your database.
• Each query holds a connection and locks rows during reads.
• Your database has maybe 200 connection slots total.
• Connection pool exhausted → requests queue → latency spikes to 2–5 seconds → users bounce → your boss calls.

The page content does not actually change every millisecond. The top-20 products list is essentially the same for the next 60 seconds. You are paying the full SQL cost for every single request even though the answer is the same. That is the problem Redis solves.

The fix: cache the result in Redis

The pattern is three lines of logic:

1. On request: try to read the pre-computed result from Redis with GET top-products.
2. Cache hit (99% of requests): return it. Time: ~1 ms. No database touched.
3. Cache miss (1% of requests, after TTL expires): run the SQL queries once, store the result with SET top-products <json> EX 60, return it.

The result in production: database load drops ~99%. User-facing latency drops from ~80 ms to ~1 ms — an 80× improvement. And your database is now handling ~100 queries per second instead of 50,000, leaving plenty of headroom for writes and other operations.

Here is the key mental shift that separates Redis from every other cache or database you have probably used.

Traditional stores: you move data, your code does the work

In most caches and databases, the server just stores bytes. You want to add an item to a list? You have to: (1) fetch the whole list as a string or JSON blob, (2) parse it into an object in your application, (3) append the new item, (4) re-serialize the whole thing back to a string, (5) write the whole string back to the server. Five round-trips of logic, and the data had to travel across the network twice.

Redis: the server does the work, you send one command

Redis stores your data in a typed structure that the server understands. You want to add an item to a list? You send one command: LPUSH my-list "new-item". Redis appends the item internally in O(1) time and sends back the new list length. No fetching. No parsing. No re-serializing. No second write. The compute happened where the data lives.

This idea — moving computation to the data instead of moving data to the computation — is why Redis stays fast even for operations that look complex. Incrementing a counter, updating a score, adding a member to a set: all atomic, all server-side, all in microseconds.

The durability side of the story

You might worry: "If Redis is all RAM, a server restart wipes everything." Redis has two durability mechanisms to prevent this. The Append-Only File (AOF) writes every command to disk as it arrives — think of it as a receipt tape. On restart, Redis replays every command and rebuilds the dataset. The RDB snapshot takes a complete binary copy of the dataset every few minutes. Most production deployments run both, giving you the speed of RAM during normal operation and the safety net of disk for recovery.

Before diving into data structures and commands, let's pin down six concepts. You will see these in every Redis discussion, every config file, and every interview. Understand these and everything else will click.

This is where Redis gets genuinely exciting. Most key-value stores let you put a string in and get a string back. Redis gives you eight server-understood data structures, each with its own set of commands optimized for that shape. Pick the right structure and your code becomes three lines instead of thirty.

Structure-by-structure plain English

See it in code

# Plain cache value with 5-minute TTL
SET product:42 '{"name":"Keyboard","price":99}' EX 300

# Read it back
GET product:42
# → '{"name":"Keyboard","price":99}'

# Atomic counter — no race conditions ever
SET page-views 0
INCR page-views   # → 1
INCR page-views   # → 2
INCRBY page-views 10  # → 12

# Rate limiting: allow 100 requests per minute per IP
# Each call increments the counter and checks it
INCR rate:1.2.3.4
EXPIRE rate:1.2.3.4 60
# If result > 100, reject the request

# Check TTL (seconds remaining)
TTL product:42   # → 287 (about 4:47 left)

# NX flag: set only if key does NOT exist (distributed lock pattern)
SET lock:resource-1 "worker-A" NX EX 30
# → OK if lock acquired, (nil) if already held

# Hash: store user profile fields individually
HSET user:42 name Alice age 30 city London login_count 0

# Read one field — no need to fetch the whole object
HGET user:42 name        # → Alice
HGET user:42 city        # → London

# Read all fields
HGETALL user:42
# → name Alice age 30 city London login_count 0

# Update one field without touching others
HINCRBY user:42 login_count 1   # → 1

# ─── Lists as a FIFO queue ───
# Producer pushes jobs to the head
LPUSH job-queue '{"task":"send-email","to":"alice@example.com"}'
LPUSH job-queue '{"task":"resize-image","id":99}'

# Worker pops from the tail (FIFO order)
RPOP job-queue   # → '{"task":"send-email",...}'

# Blocking pop: worker sleeps until a job arrives (no polling!)
BRPOP job-queue 30   # wait up to 30 seconds

# Recent activity feed: keep last 10 items
LPUSH activity:user:42 "logged in"
LTRIM activity:user:42 0 9   # trim to 10 items max
LRANGE activity:user:42 0 9  # read all 10

# Add players with initial scores
ZADD leaderboard 1500 alice
ZADD leaderboard 2300 bob
ZADD leaderboard 1800 carol

# Alice scores 200 more points
ZINCRBY leaderboard 200 alice   # alice now has 1700

# Top 3 players (high score first)
ZRANGE leaderboard 0 2 WITHSCORES REV
# → 1) bob 2300
# → 2) carol 1800
# → 3) alice 1700

# Alice's rank (0-indexed, highest score = rank 0)
ZREVRANK leaderboard alice   # → 2 (3rd place)

# Players in score range 1500–2000
ZRANGEBYSCORE leaderboard 1500 2000 WITHSCORES
# → alice 1700, carol 1800

# How many players have score >= 1600?
ZCOUNT leaderboard 1600 +inf   # → 2 (bob, carol)

# Remove a player
ZREM leaderboard alice

The most common question about Redis: "If everything lives in RAM, what happens when the server crashes or reboots?" The answer is: you decide. Redis gives you two complementary persistence mechanisms, and most production setups use both.

AOF vs RDB — the trade-off in plain terms

A single Redis node is blazing fast — but it is also a single point of failure. If that one server goes down, every cache miss hits your database at once (a thundering herd), sessions vanish, rate-limit counters reset. For any production workload that matters, you need a backup plan. Redis's answer is replication: one primary node that accepts writes, plus one or more replica nodes that copy those writes and can serve reads.

How replication works — plain English first

Think of the primary as the "source of truth" notebook. Replicas are photocopies that automatically stay in sync. When your app writes SET user:42:name "Alice", the primary executes it instantly and — asynchronously, a tiny fraction of a second later — ships that same command to every replica. The replica applies it to its own in-memory state. Reads can go to any replica; writes always go to the primary.

The word asynchronously is critical. The primary does not wait for replicas to confirm before acknowledging the write to your application. This means in the normal case replication is essentially instantaneous (typically under 100 ms on a healthy LAN), but during a crash there may be a small window of commands that reached the primary but never made it to replicas — those writes can be lost.

Three things you must know about Redis replication

Replication gives you copies of your data. But copies are useless if nobody promotes one when the original breaks. Redis Sentinel is the watchdog process that does exactly that: it monitors your primary and replicas, detects when the primary is unreachable, and orchestrates the promotion of a replica to become the new primary — automatically, while notifying your clients where to reconnect.

You run Sentinel as a separate process (usually three instances on separate machines). Each Sentinel monitors the primary independently. The key word is quorum: a single Sentinel saying "the primary is down" is not enough to trigger failover — a majority must agree. This prevents a network hiccup from one machine causing a false alarm that breaks your production cluster.

Sentinel vs Cluster — when to use which

Sentinel keeps you alive when one node dies. But what if your dataset is simply too large for one machine's RAM, or your write throughput is too high for one CPU? That is where Redis Cluster comes in. It splits your data across multiple master nodes automatically — each node owns a slice of the keyspace — and each master has its own replica(s) for HA. Sharding and replication in a single system.

Hash slots — the math behind the split

Redis Cluster divides the keyspace into exactly 16,384 hash slots. When you write a key, Redis runs CRC16(key) % 16384 to determine which slot it belongs to. Slots are then assigned across master nodes. With three masters you might assign slots 0–5460 to Master A, 5461–10922 to Master B, and 10923–16383 to Master C. Every key finds exactly one node — no guessing, no coordination needed per-request.

The number 16,384 was chosen deliberately: it is large enough that slot assignment is fine-grained (you can balance evenly across many nodes), but small enough that the slot bitmap (a 2 KB structure) fits in a heartbeat gossip message between nodes, keeping cluster overhead low.

Key concepts you must understand

Redis is in-memory — so RAM is the hard resource limit. You can configure a maximum memory usage with maxmemory 4gb. When Redis hits that limit and a new write arrives, it has to decide: refuse the write, or evict (delete) an existing key to make space. The eviction policy controls that choice. Pick the wrong policy and you get either silent data loss or production errors — both bad in different ways.

There are eight built-in policies, split along two axes: which keys are eligible (all keys, or only keys with a TTL set) and how to pick the victim (LRU — least recently used; LFU — least frequently used; random; or by remaining TTL).

The eight policies — when to use each

Redis is not just a cache. It has two distinct messaging primitives built in, and picking the right one matters a lot. The confusion between them trips up engineers regularly, so let's be very clear about the difference before any code.

Pub/Sub is like a live TV broadcast: the channel is live, and only viewers watching right now receive the message. Miss the show? You missed it. No replay, no buffer, no history. This is fire-and-forget, and that is intentional — it is incredibly fast and requires zero storage.

Streams are like a DVR recording: every message is appended to a durable log. Consumers can read from any point in time (replay), multiple consumer groups can independently track their position, and messages survive disconnections. Think of it as a lightweight Kafka built into Redis, available since Redis 5.0 (released 2018).

Three messaging patterns and when to pick each

Here is something surprising about Redis: it is fundamentally single-threaded for command execution. One command runs to completion, then the next starts. No two commands ever interleave inside Redis. This sounds like a weakness — but it is actually a superpower for atomic operations, because it means a Lua script you send to Redis runs entirely without interruption. No other client can sneak a command in between your script's steps.

Why does this matter? Imagine you need to "increment a counter, but only if it is below 100 — and do it safely even with 10,000 concurrent clients." With separate GET then SET commands there is a race condition: two clients could both read 99, both decide to increment, and both write 100. With a Lua script, that read-check-write is a single atomic unit. Nobody else touches the key while it runs.

Three patterns for atomic Redis operations

-- Increment counter only if below cap.
-- KEYS[1] = counter key, ARGV[1] = cap value
EVAL "
  local current = tonumber(redis.call('GET', KEYS[1]) or 0)
  if current < tonumber(ARGV[1]) then
    return redis.call('INCR', KEYS[1])
  else
    return current
  end
" 1 rate:user:42 100

-- Optimistic transfer: debit A, credit B.
-- WATCH lets us detect if balance changed under us.

WATCH balance:alice
GET balance:alice           -- read current balance: 200

MULTI
  DECRBY balance:alice 50   -- queued (not run yet)
  INCRBY balance:bob 50     -- queued (not run yet)
EXEC
-- If balance:alice changed between WATCH and EXEC → returns nil (retry)
-- Otherwise → both commands execute atomically

-- Step 1: Register the library (once, at deploy time)
FUNCTION LOAD "#!lua name=mylib\n
  redis.register_function('incr_cap', function(keys, args)
    local current = tonumber(redis.call('GET', keys[1]) or 0)
    if current < tonumber(args[1]) then
      return redis.call('INCR', keys[1])
    end
    return current
  end)
"

-- Step 2: Call it by name — no script source sent over the wire
FCALL incr_cap 1 rate:user:42 100

Redis as a cache is the number-one use case you will encounter. But "just throw Redis in front of the database" is not a strategy — it is a plan to introduce subtle bugs. The pattern you choose determines whether your cache stays correct, how much latency it adds on writes, and what happens when the cache restarts cold.

There are four battle-tested patterns. Each one answers a slightly different question: who is responsible for filling the cache? Who decides when to write through to the database? How much latency is acceptable on writes?

import redis
import json

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def get_user(user_id: int) -> dict:
    cache_key = f"user:{user_id}"

    # Step 1 — check the cache
    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)   # cache hit: fast path

    # Step 2 — cache miss: load from database
    user = db.query("SELECT * FROM users WHERE id = %s", user_id)
    if not user:
        return None

    # Step 3 — populate the cache for next time (TTL = 5 minutes)
    r.setex(cache_key, 300, json.dumps(user))
    return user

import random
import redis
import json

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

BASE_TTL = 300  # 5 minutes base

def get_user(user_id: int) -> dict:
    cache_key = f"user:{user_id}"
    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)

    user = db.query("SELECT * FROM users WHERE id = %s", user_id)
    if not user:
        return None

    # Add jitter: ±30 seconds so all keys don't expire at the same instant
    # Without jitter, if 1000 keys were set at the same time, they all
    # expire together — thundering herd. Jitter spreads expiry out.
    ttl = BASE_TTL + random.randint(-30, 30)
    r.setex(cache_key, ttl, json.dumps(user))
    return user

import redis
import json

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def update_user(user_id: int, data: dict) -> None:
    # Write-through: update DB first (source of truth), then cache.
    # Order matters: if the DB write fails, we don't pollute the cache.
    db.execute("UPDATE users SET ... WHERE id = %s", user_id)

    cache_key = f"user:{user_id}"
    r.setex(cache_key, 300, json.dumps(data))
    # Now both DB and cache have the fresh value.
    # Reads will hit the cache and always see up-to-date data.

Caching is how most people first meet Redis, but it is only one of eight very different jobs Redis does well. Once you understand the data structures, you start seeing Redis-shaped holes everywhere in your architecture — places where a small, fast, in-memory operation can replace an expensive database round-trip or a complex piece of application logic.

SET session:abc123 '{"user":42,"role":"admin"}' EX 1800

-- Lua script (atomic): increment + set expiry only on first call
local count = redis.call('INCR', KEYS[1])
if count == 1 then redis.call('EXPIRE', KEYS[1], 60) end
return count

PFADD page:views:2025-05-09 user:42 user:99 user:7
PFCOUNT page:views:2025-05-09   -- returns ~3 (approx unique count)

A lock prevents two processes from doing the same thing at the same time. In a single program, this is solved with a mutex — a language primitive backed by the operating system. In a distributed system spanning multiple servers, there is no shared memory, no OS mutex. You have to coordinate through a network, and networks are unreliable.

The naive Redis approach — SET lock NX EX 30 — works surprisingly well for many everyday tasks like "only one process should run this cron job." But once the stakes rise (financial operations, idempotency guarantees), three subtle failure modes appear.

-- Release lock safely (Lua — atomic check-and-delete)
if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
else
    return 0   -- someone else's lock, don't touch it
end

Redis is famously fast, but "fast" needs a number attached to it. A single Redis instance typically handles somewhere in the range of 50,000 to 200,000 operations per second on commodity hardware. That wide range reflects real variables: command type (O(1) GET vs O(N) LRANGE), payload size, network topology, and whether you are using pipelining.

Understanding why Redis is fast also tells you exactly where it can become slow — and how to avoid those traps.

For most of its life, Redis was a beloved open-source project under the permissive BSD license. In March 2024, Redis Inc. changed the license to a dual SSPL/RSALv2 model. The practical effect: cloud providers can no longer offer a managed Redis service without commercial terms. The response from the broader community was swift — a fork of the last BSD-licensed version (7.2.4) was announced under the Linux Foundation banner and named Valkey.

This is not just a legal footnote. It changes which software your cloud provider's managed service runs, which version gets community bug fixes, and what the long-term maintenance path looks like for your infrastructure.

Running Redis in a proof-of-concept is trivial: redis-server and you are done. Running Redis in production is a different story. Production Redis needs persistence tuned to your durability requirements, monitoring so you know when something is wrong before your users do, security so that it is not an open door on your network, a backup strategy, capacity headroom, and a plan for cluster operations.

None of these are exotic requirements — they apply to any stateful service. But Redis has a few specific characteristics that make each one worth understanding explicitly.

Redis has a tight, well-organised toolbox. You will reach for the command-line client for quick checks and scripting, the official GUI when you need to understand what is inside a live server, the built-in benchmark when you want raw numbers, and a language driver when your application talks to Redis. Here is the rundown of the six you will use most.

# Connect to a local Redis (default port 6379)
redis-cli

# Or connect to a remote server with auth
redis-cli -h my-redis.example.com -p 6379 -a $REDIS_PASSWORD

# Basic string ops
SET user:42:name "Rafikul"          # store a string
GET user:42:name                     # => "Rafikul"
SET page:hits 0
INCR page:hits                       # atomic increment => 1
EXPIRE page:hits 86400               # expire in 24 h

# Hash (flat object)
HSET session:abc token "xyz123" user_id 42 created_at 1715000000
HGETALL session:abc                  # returns all fields and values
HGET session:abc user_id             # => "42"

# Sorted set (leaderboard)
ZADD leaderboard 1500 "alice"
ZADD leaderboard 2300 "bob"
ZADD leaderboard 1900 "carol"
ZRANGE leaderboard 0 -1 WITHSCORES REV   # top scores, highest first

# Safe key traversal — NEVER use KEYS * in production
SCAN 0 MATCH "user:*" COUNT 100     # returns cursor + batch of keys
# loop: feed returned cursor back until cursor == 0

# Check server health
INFO server      # version, uptime, config
INFO memory      # used_memory, mem_fragmentation_ratio
INFO stats       # ops/sec, hit ratio

import redis
import os

# One client, reused across the application. Pool size defaults to 10.
r = redis.Redis(
    host=os.environ["REDIS_HOST"],
    port=6379,
    password=os.environ.get("REDIS_PASSWORD"),
    decode_responses=True,   # return str, not bytes
)

# --- Cache pattern: get-or-set with TTL ---
def get_user_profile(user_id: int) -> dict:
    key = f"user:{user_id}:profile"
    cached = r.hgetall(key)
    if cached:
        return cached                         # cache HIT

    # cache MISS — fetch from database
    profile = db_fetch_user(user_id)          # your DB call here
    r.hset(key, mapping=profile)
    r.expire(key, 3600)                       # 1-hour TTL
    return profile

# --- Sorted set: real-time leaderboard ---
def update_score(user_id: str, delta: int):
    r.zincrby("game:scores", delta, user_id)  # atomic increment

def top_10() -> list[tuple]:
    # ZRANGE with REV + WITHSCORES returns highest-score first
    return r.zrange("game:scores", 0, 9, withscores=True, desc=True)

# --- Rate limiter: sliding window via INCR + EXPIRE ---
def is_rate_limited(ip: str, limit: int = 100, window: int = 60) -> bool:
    key = f"rate:{ip}:{int(time.time()) // window}"
    count = r.incr(key)
    if count == 1:
        r.expire(key, window * 2)  # keep key slightly longer than window
    return count > limit

import Redis from 'ioredis';

// One client instance — reuse across the application
const redis = new Redis({
  host: process.env.REDIS_HOST,
  port: 6379,
  password: process.env.REDIS_PASSWORD,
  lazyConnect: true,           // don't connect until first command
  retryStrategy: times => Math.min(times * 50, 2000), // exponential back-off
});

// --- Pipeline: batch 3 writes in ONE round-trip ---
// Without pipeline: 3 × RTT. With pipeline: 1 × RTT.
async function recordPageView(userId, page) {
  const pipeline = redis.pipeline();
  pipeline.incr(`page:${page}:views`);
  pipeline.lpush(`user:${userId}:history`, page);
  pipeline.ltrim(`user:${userId}:history`, 0, 99);  // keep last 100 only
  const results = await pipeline.exec();
  // results: [[null, 42], [null, 5], [null, "OK"]]  (err, value) pairs
  return results;
}

// --- Cache-aside with TTL ---
async function getCachedProduct(productId) {
  const key = `product:${productId}`;
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);         // cache HIT

  const product = await db.getProduct(productId); // DB fallback
  await redis.setex(key, 300, JSON.stringify(product)); // 5-min TTL
  return product;
}

// --- Pub/Sub: fire-and-forget notifications ---
async function publishEvent(channel, payload) {
  await redis.publish(channel, JSON.stringify(payload));
}

// Subscriber must use a SEPARATE connection (ioredis auto-handles this)
const sub = redis.duplicate();
sub.subscribe('order:created', (err, count) => {
  if (err) console.error(err);
});
sub.on('message', (channel, message) => {
  console.log(`[${channel}]`, JSON.parse(message));
});

Redis has a reputation that is partly accurate and partly a decade-old snapshot of a very different product. The six beliefs below circulate widely in interviews and tech Twitter. Each one contains a grain of truth — which is exactly why they persist. Knowing why each one is at least partially wrong is more useful than just knowing the correct answer.

These are real patterns drawn from widely-reported production failures and recurring incidents that Redis practitioners encounter repeatedly. Every one was preventable. Read them as the most concrete possible evidence that command choice, eviction policy, and replication configuration are not abstract concerns — they are the difference between a five-minute fix and a multi-hour outage.

Everything on this page distills into eight practices. None of these are arbitrary rules — each one has a clear mechanical reason rooted in how Redis actually works. Follow them and you avoid the overwhelming majority of Redis production problems. Skip any one of them and you are relying on luck.

These are the questions engineers most commonly ask when they start working with Redis in practice — or when they are deciding whether to use it at all. Plain English first, then the nuance that matters for real decisions.

Redis does not exist in isolation. The topics below are either the databases you will be asked to compare it against, the underlying concepts that explain why Redis works the way it does, or the complementary technologies it is commonly paired with. Work through them in the order that matches where you want to go next.