Redis in depth: single thread, persistence, replication and cluster

Redis appears in many designs. So the follow-ups come fast: "Does Redis lose data?" "How do you scale Redis?" "What happens when the Redis leader dies?"

A weak answer is "Redis is in memory, so it is fast; add a cluster". A strong answer says why it is fast and what can block it. It says what you lose in a crash for each way of saving data to disk. It says that copying to followers is asynchronous. And it says how the keys are split over a cluster.

At SDE-1 level you should explain why Redis is fast and what a cache loses on a restart. At SDE-2 level you should compare the ways of saving to disk and describe a failover. At SDE-3 level you should explain how keys are split over a cluster and what a hot key does. You should also say when Redis must not be the only copy of the data.

A shop keeps today's orders on a whiteboard. Reading and writing on it is very fast, but if the power goes, the board is blank. To protect the orders, the owner can take a photograph of the board every few minutes. Or the owner can write each order in a notebook as it comes. The photograph is cheap, but the orders since the last photograph are lost. The notebook loses less, but it costs more work. Redis offers both.

What Redis is

Redis is an in-memory server of data structures such as strings, lists, sets and sorted sets (lesson 63). You get a value by its key. It is often used as a cache (lesson 57).

Why Redis is fast

1. Memory. A read from memory takes about 100 ns, and an SSD read about 150 µs (lesson 30).
2. Simple structures. Most commands do a small, fixed amount of work.
3. One thread runs one command at a time. Redis uses an event loop (lesson 52). Since only one command runs at a time, each command is atomic (lesson 63), and commands do not get in each other's way.
4. Extra threads for the network. A setting can turn on threads that read, understand and write the network data (input and output, called I/O). They are off by default. The documentation still calls Redis mostly single-threaded for running commands. Other threads do slow background work, such as freeing memory.

For a number, the Redis benchmark page shows an example run of about 180,000 SET commands per second with a median time of 0.143 ms. This is only an example, and your server will differ. We can still work out the server time per command: 1,000,000 µs / 180,180 = 5.5 µs. The Redis latency page gives about 200 µs for a round trip on a 1 Gbit/s network. In many real set-ups the network, not Redis, is the slow part.

What blocks it

Because one command runs at a time, one slow command makes everyone wait. Four common causes:

• KEYS. It lists all keys that match a pattern and visits every key. The documentation says an entry-level laptop scans 1 million keys in 40 ms. Assumption: the time grows in a straight line. Then 20 million keys take 20 x 40 = 800 ms, and for that time no read is answered.
• A huge collection. A command that returns a set with millions of members takes a long time.
• Deleting a huge key. DEL frees the memory on the main thread. UNLINK removes the key at once and frees the memory on another thread.
• A long script. A script is a short program that Redis runs as one step (lesson 63). By default Redis lets it run for 5 seconds before it answers other clients with an error.

The safe replacements are SCAN and UNLINK. SCAN visits the keys in small steps. After each step it returns a cursor, a marker that says where to continue. It can return the same key twice. The SLOWLOG command lists recent slow commands. A very large value is also slow to send, and a hot key (lesson 71) puts all its reads on one thread.

Persistence: what survives a crash

Persistence means writing data to disk so that it survives a restart. Redis has four choices:

• no persistence;
• a snapshot, which saves the whole data set to a file now and then;
• an append-only file, which adds every write to a log;
• both.

Snapshot. (Redis calls it RDB.) Each rule says "save after this many seconds if at least this many writes happened". The built-in rules are: 3,600 seconds and 1 write; 300 seconds and 100 writes; 60 seconds and 10,000 writes. Redis saves with a fork: it makes a copy of the running program, called the child. The child writes the file while the main program goes on. Parent and child share memory, and a piece is copied only when one of them changes it. The documentation says to expect to lose the latest minutes of data after a crash.

The fork pauses the main thread. The system must copy the table that maps the memory of the program, and that takes longer for more memory. The Redis latency page lists about 80 ms for 6.1 GB on one physical machine (13.1 ms per GB). On an old type of virtual machine it lists 1,460 ms. These are old measurements. Your machine may be slower or faster, so measure it: the INFO command shows the last fork time as latest_fork_usec.

Append-only file. (Redis calls it AOF.) A sync forces the data from the memory of the operating system onto the disk. There are three sync settings (appendfsync):

• always: sync after each write. Very slow and the safest.
• everysec: sync once a second. This is the default, and you can lose about 1 second of writes. A slow disk can make this a little longer.
• no: leave it to the operating system. On Linux this is often 30 seconds.

The log grows, so Redis rewrites it in the background into the shortest list of commands that rebuilds the data. A counter that was increased 100 times needs 1 entry, not 100. By default the append-only file is off, and snapshots follow the built-in rules. The documentation advises using both when you want safety near that of a database such as PostgreSQL.

Replication

Redis copies from a leader to followers. (The Redis documentation says master and replica.) The copy is asynchronous by default (lesson 65). A new follower first gets a snapshot, then the stream of changes. Followers refuse writes by default, and they can serve reads that are a little old (lesson 66).

The WAIT command makes a client wait until a number of followers confirm a write. It does not make a write safe. The documentation says that confirmed writes can still be lost in a failover.

One danger: a leader with no persistence that restarts by itself comes back empty. Its followers then copy the empty data set and also lose everything.

Failover

Sentinel is a set of small separate processes that watch Redis. They check the leader, tell people when something is wrong, start a failover, and tell clients the address of the current leader. The steps are those of lesson 67:

1. The Sentinels check the leader by sending small messages. If it does not answer for the time you set, a Sentinel marks it down.
2. A set number of Sentinels agree that it is down.
3. More than half of all Sentinels must vote to let one of them run the failover.
4. That Sentinel promotes a follower. The other followers copy the new leader.
5. Clients ask the Sentinels for the new address.

The documentation advises at least three Sentinels, on machines that fail independently.

Writes that the promoted follower did not yet have are lost (lesson 67). There is also split brain (lesson 67). A client on the same side of a network break as the old leader keeps writing to it. Those writes are lost when it comes back as a follower.

The setting min-replicas-to-write limits this loss. Its name says "replicas" where this lesson says "followers". With a value of 1, a leader that cannot reach one follower stops accepting writes after a time limit you set. The cost is that if every follower is down, the leader refuses all writes. A managed service can run these steps for you.

Redis Cluster

When one machine is too small for the memory or the traffic, Redis Cluster splits the keys. It has 16,384 hash slots. The slot of a key is CRC16(key) mod 16,384, where CRC16 is a hash function (lesson 69). Each leader owns a range of slots. The client keeps a slot map, a table of which node owns which slots. If it asks the wrong node, the node answers with an error that names the right one (MOVED), and the client updates its map.

To add a node, you move some slots to it. The cluster keeps working while slots move. A cluster needs at least 3 leaders, and the documentation recommends 3 leaders and 3 followers.

This is a fixed number of buckets that an explicit table assigns to nodes. It is not the ring of lesson 70, where server positions on a circle decide the owner. A slot is the unit that moves.

Hash tags. If a key contains {...}, only the text inside is hashed. The keys {user:1000}.name and {user:1000}.cart land in the same slot. A command that uses several keys works only when all keys are in one slot. No operation works across slots.

Cluster failover. The leaders agree that a leader failed (more than half of the leaders), and one of its followers is promoted. No Sentinel is needed. Copying is still asynchronous, so writes can be lost. If a leader fails and has no follower, its slots have no owner. By default the cluster then stops answering.

Memory is the cost

The memory you need is keys x bytes per key. A follower holds the same data, so it doubles the total. The maxmemory setting limits the size. The eviction policy decides what happens when it is full (lesson 59). The policy noeviction (the default) refuses writes. The policy allkeys-lru removes the least recently used of all keys. The policy volatile-lru removes only keys that have a TTL. The command MEMORY USAGE key shows the real size of one key.

A hot key stays on one node

A cluster spreads keys, not requests (lesson 71). A hot key lives in one slot, on one node, however big the cluster is. The fixes are the same: a short-lived copy in each application server, or one key split into several names that land in different slots. A hash tag can undo this: keys that share a tag all go to one node.

Cache or database?

Ask three questions:

1. Can the data be rebuilt from another place? If yes, Redis is a cache.
2. Is the last second of writes important?
3. Do I need to find data by anything except the key?

Settings that make Redis a store: the append-only file every second, followers with Sentinel or a cluster, min-replicas-to-write, and backups copied off the machine. Even then, copying is asynchronous, so a failover can lose the latest writes. The whole data set must fit in memory. And you cannot ask "find all users with this city". Keep money and orders in a database. Use Redis for data you can rebuild or can afford to lose a little of.

# Not run here. Commands for redis-cli.
SCAN 0 MATCH sess:* COUNT 1000     # one step; repeat with the new cursor until it is 0
UNLINK report:2026                 # free a huge key without blocking
CONFIG GET save                    # show the snapshot rules
CONFIG SET appendfsync everysec    # sync the log once a second
SLOWLOG GET 5                      # the 5 most recent slow commands
CLUSTER KEYSLOT sess:1001          # 8793
CLUSTER KEYSLOT {user:1000}.name   # 1649
CLUSTER KEYSLOT {user:1000}.cart   # 1649

The slot numbers were worked out with the CRC16 code in the cluster specification. The same code gives 11058 for somekey and 2515 for foo{hash_tag}, the values in the Redis documentation of CLUSTER KEYSLOT.

A food app keeps sessions (lesson 51) in Redis at the 8 pm peak.

Assumptions (made up to teach with): 20 million sessions; 2,000 writes per second; one follower for each leader; 320 bytes for each session. The 320 bytes are a 40-byte key, a 200-byte value and 80 bytes of overhead (the extra bytes Redis keeps for each key).

Memory. 20,000,000 x 320 bytes = 6,400,000,000 bytes = 6.4 GB. With one follower: 12.8 GB.

Crash loss.

• Built-in snapshot rules: at 2,000 writes per second, the rule "60 seconds and 10,000 writes" fires every time (2,000 x 60 = 120,000 writes). So Redis saves about once a minute. A crash just before a save loses up to 60 x 2,000 = 120,000 writes.
• A slower rule that a team might choose to save work, one snapshot every 5 minutes: 5 x 60 = 300 seconds. 2,000 x 300 = 600,000 writes.
• Append-only file, synced every second: about 2,000 x 1 = 2,000 writes. This answers the predict box.

For sessions, this is fine: a user logs in again. For payment records it is not.

Fork pause. Take one instance of 6.4 GB. The old table of the Redis latency page gives 13.1 ms per GB for one machine. So 6.4 x 13.1 = 84 ms, and no command runs during that time. Measure your own machine.

Cluster of 3 leaders. 16,384 slots split into three ranges: 0 to 5,460, 5,461 to 10,922 and 10,923 to 16,383 (5,461 + 5,462 + 5,461 = 16,384). Each leader holds 6.4 / 3 = 2.13 GB, and each fork pauses 2.13 x 13.1 = 28 ms. Each leader now takes about 2,000 / 3 = 667 writes per second. The key sess:1001 is in slot 8,793, which is in the second range, so it lives on the second leader. Keys with the tag {user:1000} share slot 1,649 and live on the first leader.

A cluster of 3 leaders splits 16,384 slots into three ranges. Leader 1 owns slots 0 to 5,460, leader 2 owns 5,461 to 10,922 and leader 3 owns 10,923 to 16,383. Each leader has a follower that copies it. The key sess:1001 is in slot 8,793, so it lives on leader 2. The keys with the tag user:1000 are in slot 1,649 and live on leader 1.

Failover. Assumption: the followers are 50 ms behind. If one leader takes all 2,000 writes per second, 2,000 x 0.05 = 100 writes are lost when it dies. In the 3-leader cluster, a dying leader takes about 667 writes per second, so 667 x 0.05 = 33 writes are lost. Users on those sessions log in again.

• Snapshots solve cheap, compact backups with fast restarts. They cost a minute or more of writes (120,000 with the built-in rules, 600,000 with a 5-minute rule). I would change to the append-only file when losing that much is not acceptable.
• The append-only file solves most of that: 2,000 writes at risk. It costs a bigger file, a slower restart and more disk work. I would use both when the data matters.
• Asynchronous followers solve fast writes. They cost lost writes in a failover (100 for one leader, 33 for each leader in the cluster). I would change to min-replicas-to-write, or to a database, when no loss is allowed.
• A cluster solves the limit of one machine's memory and traffic. It costs no operation across slots and a hot key that stays on one node.

"With the append-only file, Redis never loses data." With everysec you can lose about 1 second. A failover to an asynchronous follower can lose more.

"A cluster spreads load evenly." It spreads keys. A hot key is in one slot, and a shared hash tag puts keys together on one node.

"WAIT makes a write safe." WAIT only counts confirmations from followers. The documentation says acknowledged writes can still be lost in a failover.

1. Name the four persistence choices and say what each loses in a crash. What are the 16,384 slots for?
2. A cluster holds 10 million keys of 500 bytes each, on 4 leaders with one follower each. (a) How much data is that? (b) How much does each leader hold, and how much memory is used in total? (c) With an even split of 4,096 slots each, which leader owns slot 8,793?
3. A team runs one Redis leader with two followers, with Sentinel watching, and no persistence. A tool restarts the Redis program by itself after a crash. The leader crashes. What happens to the data, and what do you change?