Case study: Metrics and logging pipeline

A company with many servers needs a place to see what they are doing. The interviewer asks: "Design the system that collects metrics and logs from 5,000 servers, stores them, and lets engineers search them. It must keep working when the servers it watches are in trouble."

A weak answer says "each server writes logs to a database". It does not count the data. It does not say what happens when the database is slow, or when the network breaks. A strong answer puts a small program on each server. It sends data in batches through a queue, and stores numbers and text in different ways. It also says how long to keep each kind, and what that costs.

At SDE-1 level you should draw the flow from server to dashboard. At SDE-2 level you should add the data sizes and the retention. At SDE-3 level you should also explain a burst, and how one wrong label can break the system.

Think about the control room of a railway. Every station sends two things. One is a short report of numbers every few seconds: trains passed, delay in minutes. The other is the stationmaster's detailed notes. A courier at each station carries both in bundles. A mail room in the control room receives the bundles. Clerks draw the numbers on charts and file the notes in cabinets. Old notes go to a cheap warehouse. Only yearly summaries of the charts are kept for many years.

This is a metrics and logging pipeline. From lesson 91: a metric is a number that we measure again and again and store with its time. A log is a record of one event. Now we build the system that moves them.

The flow and the goals

1. An agent is a small program on each server. It reads the log files and collects the metrics.
2. Receivers are servers that accept the agents' data. This step is called ingestion.
3. A queue with partitions (lesson 59) holds the data until a writer takes it.
4. Writers store metrics in a time-series database and logs in a log store.
5. A query service answers dashboards and searches.

Goals (the numbers are assumptions):

• Data is visible in dashboards within 30 seconds for 99% of the samples.
• A burst of logs, for example in an outage, must not slow the metrics.
• Metrics keep full detail for 15 days, then 5-minute points for 90 days and hourly points for 365 days. Logs are searchable for 7 days, and kept cheaply for 30 days.

The agent

The agent must never hurt the server that it watches. So:

• It batches: it sends one request every second, with all the lines and samples of that second. One request for each line would be far too many requests.
• It compresses the batch, because text compresses well.
• It retries with backoff and jitter (lesson 54) when a receiver does not answer.
• It keeps a buffer on the disk. If the network or the pipeline is down, the agent stores the data and sends it when the pipeline is back. When the buffer is full, it drops the least important lines first: the DEBUG lines (lesson 91). It never blocks the application.

Push or pull

In our design the agent pushes: it sends the data to the receivers. In a pull design, a collector asks each server for its metrics over HTTP, for example every 15 seconds. Prometheus pulls (see Sources). A pull that gets no answer shows that a server is down. A short job can end before the next pull, so it must push. Many systems use both.

Ingestion and the queue

Receivers keep no data of their own. They check a batch, add labels such as the team name, and write it to the queue. So they sit behind a load balancer (lesson 28), and you add more when the load grows.

Why a queue? It absorbs a burst, because the writers can be slower for a while (lesson 58). It lets several readers take the same data. The store writer, an alert checker and an archive job each read it with their own offsets (lesson 59). It also lets you replay the data after a bug.

Use two topics: one for metrics and one for logs. A flood of logs then cannot delay the metrics that alerts depend on (lesson 63, priority). Use the server name as the key, so the lines of one server stay in order (lesson 59).

Storing metrics: a time-series database

A sample is one value of one series at one time. A time-series database is a store made for samples. It works well because:

• Writes only add new samples to the end. The newest samples stay in memory, with a log on disk for safety, and a background job writes them to files that never change. This is the idea of lesson 46.
• The samples of one series are stored together. The times are almost regular, and the values change slowly, so they compress well. Prometheus reports 1 to 2 bytes for each sample on average (see Sources).
• An index maps each label value to the series that have it. It works like the search index of lesson 80.

The number of series is the cardinality (lesson 91). Each active series takes memory. We assume 4 KB for each series.

Storing logs

A log line is text. There are two ways to make it searchable:

• Index every word. A search for any word is fast. The index is big, and writing it costs CPU.
• Index only a few labels, such as the service, the server, the level and the time. Search finds the right chunks with the labels, then scans them. Grafana Loki works like this (see Sources). It stores far less, and it is slower for a word that appears anywhere.

Query, rollup and retention

Retention is how long data is kept. Old data is read rarely, so it needs neither full detail nor fast disks:

• Hot storage is fast disk for recent data. Cold storage is cheap object storage (lesson 24) for older data.
• A rollup (also called downsampling) replaces many old samples by fewer points: for example, one point for every 5 minutes. A rollup of a percentile must add histogram bucket counts. It must not average percentiles (lesson 91).

A dashboard that adds up 5,000 servers is slow if it reads every series. A job that precomputes the sum every minute makes it fast.

The query service picks the store for each question. Recent data comes from hot storage, and older data from rollups or cold storage. It caches the answers of popular dashboard queries for a few seconds (lesson 31).

We size the pipeline for a company with 5,000 servers.

Assumptions (made up, not real data):

• Each server sends 200 metric series, one sample every 15 seconds, and writes 50 log lines of 500 bytes per second.
• Logs compress 5 to 1. A sample takes 2 bytes. A receiver handles 1,000 batches per second. A log writer handles 20,000 lines per second.
• The peak is 3 times the average (lesson 14).
• Prices: fast disk Rs 10 for each GB each month, and object storage Rs 2. The search servers scan 10 GB per second in all.

Step 1: the volume.

• Series: 5,000 x 200 = 1,000,000. Samples per second: 1,000,000 / 15 = 66,667. Per day: 66,667 x 86,400 = 5,760,000,000 samples. At 2 bytes: 11.52 GB a day.
• Lines per second: 5,000 x 50 = 250,000. Bytes: 250,000 x 500 = 125 MB per second. Per day: 125 x 86,400 = 10,800,000 MB = 10.8 TB. Stored after compression: 10.8 / 5 = 2.16 TB = 2,160 GB a day.
• This answers the predict box: 2,160 / 11.52 = 187.5. The logs are about 190 times bigger than the metrics.
• An ordinary table with 40 bytes per sample (an assumption) would take 5,760,000,000 x 40 = 230.4 GB a day, 20 times more.

Step 2: the agent and the receivers.

• One server writes 125 MB / 5,000 = 25 KB of logs per second. A disk buffer of 1 GB (our choice) holds 1,000,000 KB / 25 KB = 40,000 seconds, about 11 hours.
• Batches: 5,000 per second, because each server sends one a second. Compressed, a batch is 25 / 5 = 5 KB, so the network carries 5,000 x 5 KB = 25 MB per second.
• Receivers: peak 3 x 5,000 = 15,000 batches per second. 15,000 / 1,000 = 15, plus 1 spare = 16.

Step 3: the queue.

• Log writers: peak 3 x 250,000 = 750,000 lines per second. 750,000 / 20,000 = 37.5, so 38 writers. A consumer group of 38 writers needs at least 38 partitions (lesson 59). We choose 64, which allows 64 x 20,000 = 1,280,000 lines per second.
• The queue keeps 6 hours of logs (our choice). Compressed: 25 MB per second x 21,600 seconds = 540,000 MB = 540 GB, for one copy.

Step 4: a burst. An outage makes the servers write 10 times more lines for 5 minutes (300 seconds). We run 40 writers: 40 x 20,000 = 800,000 lines per second.

• Arrival: 250,000 x 10 = 2,500,000 per second. The backlog grows by 2,500,000 - 800,000 = 1,700,000 per second. After 300 seconds: 1,700,000 x 300 = 510,000,000 lines.
• The newest line waits 510,000,000 / 800,000 = 637 seconds, which is 10.6 minutes. After the burst, the queue empties in 510,000,000 / (800,000 - 250,000) = 927 seconds, or 15.5 minutes.
• The metrics have their own topic, so alerts are not late.
• Drop DEBUG lines at the agent. Assume that some services keep DEBUG on, and that DEBUG is 40% of all lines. Arrival becomes 0.6 x 2,500,000 = 1,500,000. The backlog grows by 700,000 per second, to 210,000,000 lines. The newest line waits 210,000,000 / 800,000 = 262 seconds, or 4.4 minutes.
• More writers help only up to the partitions. With 64: 1,280,000 per second, backlog 1,220,000 x 300 = 366,000,000, wait 286 seconds.

Step 5: metrics storage and rollups.

• Raw, 15 days: 11.52 x 15 = 172.8 GB.
• Memory: 1,000,000 series x 4 KB = 4 GB.
• A point every 5 minutes: 288 a day for each series. A rolled-up point keeps the average and the maximum, which is 4 bytes (an assumption). 1,000,000 x 288 x 4 = 1.152 GB a day. For 90 days: 103.7 GB.
• A point every hour: 24 a day. 1,000,000 x 24 x 4 = 96 MB a day. For 365 days: 35 GB.
• All tiers: 172.8 + 103.7 + 35 = 311.5 GB. Raw for a year would be 11.52 x 365 = 4,205 GB, 13.5 times more.

Step 6: queries.

• A dashboard shows the p99 of one route for 6 hours across 5,000 servers, with 12 histogram buckets. Series: 12 x 5,000 = 60,000. Samples: 6 hours x 240 per hour = 1,440 for each series. Total 60,000 x 1,440 = 86,400,000 samples.
• With a precomputed sum for each minute: 12 series x 360 minutes = 4,320 samples. That is 86,400,000 / 4,320 = 20,000 times fewer.
• A log search over one day scans all of 2,160 GB: 2,160 / 10 = 216 seconds. Assume 200 services of equal size. With the label service, the scan reads 2,160 / 200 = 10.8 GB, which takes 1.08 seconds.

Step 7: retention and cost of logs. Assume 7 days on fast disk and 23 more days in object storage.

• Fast: 2,160 GB x 7 = 15,120 GB x Rs 10 = Rs 151,200 a month. Cold: 2,160 x 23 = 49,680 GB x Rs 2 = Rs 99,360. Total Rs 250,560.
• All 30 days on fast disk: 64,800 GB x Rs 10 = Rs 648,000, 2.6 times more.
• Reduce at the agent. Assume DEBUG is 40%, INFO 50% and WARN and ERROR 10% of the lines. Drop DEBUG and keep 1 INFO line in 5: 0.1 + 0.5 x 0.2 = 0.2. The logs shrink to 2,160 x 0.2 = 432 GB a day.
• New cost: fast 432 x 7 = 3,024 GB x 10 = Rs 30,240. Cold 432 x 23 = 9,936 GB x 2 = Rs 19,872. Total Rs 50,112, five times less.

Step 8: the twist. A team adds a customer_id label to the request counter. That counter has 5,000 servers x 20 routes x 5 statuses = 500,000 series. Each server sees 50 customers.

• New series: 500,000 x 50 = 25,000,000. The total becomes 1,000,000 - 500,000 + 25,000,000 = 25,500,000. That is 25.5 times more.
• Memory: 25,500,000 x 4 KB = 102 GB, against 4 GB. Samples: 25,500,000 / 15 = 1,700,000 per second, against 66,667.
• The fix: remove the label. Logs and traces answer per-customer questions (lesson 91). Receivers also set a limit on the series of each metric, drop the new series above it, and count the drops. To count the series of a metric, a receiver does not keep a list of 25,000,000 ids (200 MB at 8 bytes each). It keeps a HyperLogLog (lesson 82), which needs about 12 KB.

Freshness. Assume these times: batch 1 s, network 0.2 s, receiver 0.1 s, queue 1 s, writer 5 s, index 5 s. The sum is 12.3 seconds, inside our 30 seconds. In the burst of Step 4 the logs are late by minutes, but the metrics are not. The pipeline must watch itself. Alert on the lag of the queue (lesson 59) and on the agent buffers, from outside the pipeline.

• A queue solves bursts, and gives replay and many readers. It costs storage (540 GB for 6 hours) and delay (10.6 minutes in the burst). I would keep metrics and logs in separate topics.
• Indexing only labels solves the size of the index. It costs slower searches for a word (216 seconds without a label). I would index every word only for services that need fast text search.
• Dropping lines at the agent solves cost: Rs 50,112, not Rs 250,560. It costs lost lines, which may explain a rare bug. I would drop DEBUG first and sample only routine lines.
• Rollups solve the size of old metrics: 311.5 GB, not 4,205 GB. They cost detail. I would keep the raw data for the days when people look at it.

"Each server should write its logs and metrics straight to the database." At 250,000 lines per second, the database is the bottleneck. A slow database blocks the servers. Batches, an agent buffer and a queue protect both sides.

"Metrics and logs are the same kind of data, so one store is fine." Logs are 190 times bigger than metrics here. They need different storage, retention and indexing.

"More labels on a metric give more information for free." Each new combination is a new series. One label with 50 values per server turned 1,000,000 series into 25,500,000, and memory from 4 GB into 102 GB.

1. Why does the agent keep a buffer on the disk? Why are metrics and logs in different topics? What is cardinality, and why does it matter?
2. 2,000 servers each write 100 log lines per second of 400 bytes. Compression is 4 to 1. (a) How many GB are stored in a day? (b) With 7 days on fast disk at Rs 10 for each GB each month, what is the monthly cost? (c) A writer handles 25,000 lines per second. How many writers are needed at a peak of 3 times the average?
3. A metric has 300 series. A developer adds a label order_id, and 1,000,000 different orders appear each day. Each order makes at least one new series. What happens, and what do you do?