Fan-out on write vs fan-out on read

The decision

A feed is the list of newest posts from the accounts a user follows. Fan-out means that one post spreads to many users. With fan-out on write (push), the system adds the post ID to every follower's feed list when the post is made. Opening a feed is then one lookup. With fan-out on read (pull), the system does nothing at posting time. When a user opens the feed, it fetches recent posts from every followed account and merges them. Push moves the work to the writer's side. Pull moves it to the reader's side. The numbers of posts, feed opens and followers decide which is cheaper.

Choose fan-out on write when

• Feed opens far outnumber posts. In the worked example below the ratio is 20 to 1.
• Most accounts have a modest number of followers, so one post means a few hundred list writes.
• The feed must open fast, in tens of milliseconds.
• You can keep the feed lists in memory for active users.

Choose fan-out on read when

• A few accounts have millions of followers (stars). Pushing their posts is a huge burst of writes.
• Users follow very many accounts, or most followers are inactive and would never read the pushed post.
• The feed is ranked at read time with fresh signals, so a pre-built list would be reordered anyway.
• Edits, deletes and privacy changes must show at once, and you do not want to hunt through millions of lists.

The numbers that change the answer

Worked example. Assumptions: 10,000,000 (1 crore) daily active users. Each user opens the feed 10 times a day, follows 200 accounts and has 200 followers on average. The app receives 5,000,000 (50 lakh) posts a day.

Work per day (calculation).

• Push list writes = posts x followers = 5,000,000 x 200 = 1,000,000,000 per day. Per second: 1,000,000,000 / 86,400 = 11,574. At a peak 3 times the average (assumption), that is 34,722 per second.
• Pull lookups = feed opens x accounts followed = (10,000,000 x 10) x 200 = 20,000,000,000 per day. Per second: 231,481.
• Pull does 20,000,000,000 / 1,000,000,000 = 20 times more lookups than push does writes. Push wins here because reads are 20 times more common than posts.

Feed open time (assumption plus calculation). A cache read takes 0.3 ms. Pull with 200 accounts and 20 parallel reads needs 200 / 20 = 10 rounds. That is 10 x 0.3 ms = 3 ms plus merge time, if nothing is slow. Push needs one list read.

The slow tail (calculation). Assumption: 1% of lookups are slow. Pull waits for the slowest of 200. The chance that all 200 are fast is 0.99 to the power of 200 = 0.134. So 1 - 0.134 = 86.6% of pull feed opens meet at least one slow lookup. Push does one list lookup, so about 1% meet a slow list lookup. Fetching the post text adds lookups, but they run together.

The star (calculation). One account has 10,000,000 followers and posts once. Assumption: one list server adds 35,000 entries per second.

• One server: 10,000,000 / 35,000 = 286 seconds, nearly 5 minutes.
• 10 servers in parallel: 28.6 seconds.

The celebrity threshold (calculation). A push must reach all followers within the freshness target, using spare capacity. Assumptions: spare list writes 20,000 per second, target 60 seconds. Threshold = 20,000 x 60 = 1,200,000 followers. Accounts above it are pulled. Accounts below are pushed.

Memory (calculation). A list keeps 200 post IDs of 8 bytes each = 1,600 bytes. For 10,000,000 active users that is 16,000,000,000 bytes = 16 GB, or 48 GB with 3 copies. This is the raw size, and a real store adds overhead. Pushing to 100,000,000 registered users would need 160 GB, so skip users inactive for 30 days.

What breaks with each choice

Fan-out on write.

• A star post creates a burst of writes, and the posts of normal accounts wait behind it in the queue. Fallback: a priority lane for big accounts, or treat stars as pull.
• A deleted post or an unfollow leaves IDs in many lists. Fallback: skip deleted IDs at read time, and clean lists in the background.
• Workers crash in the middle of a fan-out job and run it again. Fallback: make the insert idempotent, so adding the same ID twice changes nothing.
• The queue falls behind. Fallback: insert into the author's own list first, and alert on queue age.

Fan-out on read.

• Read cost grows with the number of followed accounts, and most opens meet a slow lookup. Fallback: a time limit per lookup, return a partial feed, and send duplicate requests to a second cache copy after a short wait.
• A popular account's recent posts are read by millions, so one cache key is hot. Fallback: copy that key on several cache servers.

Hybrid (the usual answer). Push for normal accounts, pull for stars, merge at read time. It has both sets of failure modes, so you pay for two code paths and one merge step.

What it costs

• Servers. Push needs capacity for 34,722 list writes per second at peak and 48 GB of memory. Pull needs a cache tier for 231,481 lookups per second.
• People. Hybrid needs a threshold, a queue with two lanes, a merge step and a rebuild job. Assumption: 4 to 8 engineer-weeks.
• Changing your mind. This is a cheap and reversible choice because lists are derived data. You can rebuild a list by pulling. Move the threshold one step at a time and watch queue age and cache load. Assumption: a few days per step.

How each level answers

SDE-1: Push builds the feed when someone posts. Pull builds it when someone opens the app. Push makes reading fast.

SDE-2: I push for normal accounts, because people read much more than they post. I pull for accounts with millions of followers, because one post would make millions of writes.

Senior: With 5,000,000 posts and 200 followers, push is 11,574 writes per second against 231,481 lookups per second for pull. A star with 10,000,000 followers takes 286 seconds on one list server, so I set the threshold near 1,200,000 followers and use a priority lane. For pull I expect 86.6% of opens to meet a slow lookup, so I set time limits and return a partial feed.

Staff: The business question is how fast a post must appear and how fast the feed must open. I ask for those two targets, then set the threshold from them, not from habit. I would ship push first for normal users, add the star path when the first big account arrives, and keep the choice behind a flag. I measure feed open time at the 99th percentile, the age of the oldest fan-out job and the memory per user. Product owns the freshness target. The feed team owns the threshold and the rebuild job.

Say it in the interview

Reads are 20 times more common than posts, so I would push post IDs into follower lists. That is about 11,600 list writes per second, and a feed open is one lookup. Accounts above roughly a million followers would be pulled and merged at read time, because pushing one star post takes minutes on a single worker.

Common mistakes

• Choosing one method for everyone. A star breaks push, and a normal user breaks pull on cost. The hybrid is the answer to the follower count, not to the average.
• Quoting only the daily average. The average hides the burst: one star post is 10,000,000 writes at once.
• Pushing to inactive users. Those writes are never read and use memory. Skip users inactive for 30 days and rebuild when they return.
• Storing whole posts in the lists. Store 8-byte IDs. A post copied 10,000,000 times is costly to edit.