Trace sampling vs full capture

The decision

A trace is the record of one request as it moves through many services. It is made of spans, one for each step. Storing every span of every request is expensive. You have three options. Full capture keeps every trace. Head sampling decides at the start of the request, for example "keep 1 in 100". It decides before anyone knows if the request will be slow or fail. Tail sampling waits until the whole trace is finished, then keeps the ones that matter: errors, slow requests and a small random share. The forces are money, the chance of missing the one trace you need, and the work to run the sampler.

Choose full capture when

• Traffic is low. The Dapper paper from Google says that services with perhaps dozens of requests a second can afford to trace every request. It contrasts them with tens of thousands (see Sources).
• Rules or customers need a record of every request, for example for audits.
• The trace data is short-lived, kept for hours, so the storage stays small.

Choose head sampling when

• Traffic is high and most requests are healthy and alike.
• You want the simplest setup, with no memory held waiting for traces to finish.
• The problems you chase happen often, so some sampled traces will show them.
• You need a bounded bill that you can predict.

Choose tail sampling when

• You must keep nearly every error and every slow request.
• Bugs hit few users, such as one customer or one rare path.
• You can run collectors that hold traces in memory, and send all spans of one trace to the same collector.
• The savings are worth the extra moving part. The OpenTelemetry documentation says tail sampling needs stateful systems that store a large amount of data (see Sources).

The numbers that change the answer

Assumptions: 2,000 requests a second, 12 spans in each trace, and 500 bytes for each span. The Dapper paper reports 426 bytes on average for each span in its repository, so our figure is near that (see Sources). Ingest and storage cost Rs 40 per GB (Assumption).

Volume (calculation).

• Traces a day = 2,000 x 86,400 = 172,800,000.
• Spans a day = 172,800,000 x 12 = 2,073,600,000.
• Bytes a day = 2,073,600,000 x 500 = 1,036,800,000,000 = about 1,037 GB, or 1.04 TB.
• A month (30 days) = 1,037 x 30 = 31,104 GB.

Cost a month (calculation, price is an assumption).

The tail share is an assumption: 1% of requests are slow, 0.01% fail, and we add a random 0.5% for a baseline. That is 1 + 0.01 + 0.5 = 1.51%, so about 1.5%.

The missed bug (calculation). One customer has a bug that touches 50 requests a day. The chance that head sampling keeps at least one of those traces is 1 minus the chance of keeping none.

• At 1%: 1 - 0.99 to the power of 50 = 39.5%.
• At 0.1%: 1 - 0.999 to the power of 50 = 4.9%.
• At 10%: 1 - 0.9 to the power of 50 = 99.5%.
• With tail sampling, if the bug makes the request fail or run slowly, every one of the 50 is kept.

For common errors head sampling is fine. At 2,000 requests a second and a 0.01% error rate, there are 172,800,000 x 0.0001 = 17,280 failed requests a day. At 1%, you keep 173 of them. That is plenty to find a cause that repeats.

Memory for tail sampling (calculation). Assumption: the sampler waits 30 seconds for a trace to finish. It holds 2,000 x 30 = 60,000 traces, or 60,000 x 12 = 720,000 spans, or 720,000 x 500 = 360,000,000 bytes = 360 MB. This is small, but spans held in memory take more room than their stored size. The hard part is that all spans of one trace must reach one collector. So you need routing by trace ID and enough collectors for bursts.

Cited facts (Dapper paper, 2010). In Google's tracing system, a sampling rate as low as 1 in 1,024 still left an adequate amount of trace data for high-volume services. They write that sampling is needed to avoid noticeable latency loss on a web search workload. They also write that lower-traffic workloads may miss important events at such low rates (see Sources). These are results from Google's own system in 2010, so test them on yours.

What breaks with each choice

Full capture.

• The bill grows with traffic (Rs 1.24 million a month in the example), and a traffic spike may overload the trace backend. Fallback: a hard cap, and a drop rule for the busiest, healthiest routes.
• Tracing adds latency on the busy path. Fallback: send spans asynchronously and in batches.

Head sampling.

• The rare and important trace is dropped before you know it matters (39.5% chance of a trace for the 50-request bug at 1%). Fallback: a higher rate on important routes, and a flag that forces tracing for one user or one request.

Tail sampling.

• A collector crash loses the traces it held. Fallback: accept some loss, or run in pairs.
• Collectors fall behind in a burst and drop data. OpenTelemetry notes that a tail sampler may need to fall back to simpler sampling if it cannot keep up. Fallback: alert on dropped traces.

What it costs

• Money: Rs 1,244,160 a month for full capture against about Rs 18,662 for tail sampling in the example (both calculated from our price assumption). The saving is 1,244,160 - 18,662 = Rs 1,225,498, about Rs 1.2 million a month. Collector servers cost extra.
• People: Assumption: head sampling takes 1 engineer-week to set up. Tail sampling takes 4 to 6 engineer-weeks and some on-call work.
• Reversible-first path. Start with full capture in test and for low-traffic services. In production begin with head sampling at 10%, and a way to force tracing for one request. Keep error rates and latency in metrics, which are cheap and complete. Move to tail sampling when your team says "we could not find the trace" more than twice a month (Assumption: our trigger). Move also when the bill passes the cost of the extra collectors.
• Changing your mind: changing the head rate is a configuration change in minutes. Moving to tail sampling is a project of weeks. Moving back is easy.

How each level answers

The question is: "Design tracing for 2,000 requests a second."

SDE-1: A trace follows a request across services. Sampling keeps only some traces to save space.

SDE-2: I sample 1% of requests at the start and pass the decision along, and I count all requests and errors in metrics. This keeps cost low.

Senior: Full capture is about 1,037 GB a day, Rs 1.24 million a month. A 1% head sample is Rs 12,442, but it has a 39.5% chance of showing a bug that touches 50 requests. So I tail-sample: keep errors, slow requests and 0.5% at random, about 1.5% in all. The sampler holds about 360 MB. The fallback is a forced-trace flag.

Staff: I judge it by the cost of a longer incident when the trace is missing, against a bill of Rs 1.2 million a month. I make the platform team own collectors and the rules, and service teams own their custom attributes. I measure the share of incidents where the trace was found, dropped traces and cost per million requests. The path is head sampling, then tail sampling for the routes that matter. I revisit it when traffic doubles or the bill changes.

Say it in the interview

At 2,000 requests a second, full capture is about 1 TB a day. I would keep metrics for everything and sample traces. Tail sampling would keep every error and slow request plus 0.5% random, about 1.5% of the data. A 1% head sample alone has under a 40% chance of keeping a trace for a bug that touches 50 requests a day.

Common mistakes

• Sampling the metrics too. Counts and error rates must cover all requests. Only the traces are sampled.
• Choosing the rate by the bill alone. Ask which questions the traces must answer and what one missed trace costs.
• Forgetting the forced-trace switch. Support needs to trace one customer's request on demand.

Sources

• Sigelman, Barroso, Burrows, Stephenson, Plakal, Beaver, Jaspan and Shanbhag, "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure", Google Technical Report dapper-2010-1, April 2010. Section 4.3: sampling is needed to avoid noticeable latency loss, and an adequate amount of trace data remains for high-volume services at a rate as low as 1/1024. Section 4.4: the first production version averaged one sampled trace in 1024 candidates, and lower-traffic workloads may miss important events at such rates. Section 4.5: services with perhaps dozens rather than tens of thousands of requests a second can afford to trace every request. Section 4.2: 426 bytes on average for each span in the repository. https://research.google.com/archive/papers/dapper-2010-1.pdf (opens in a new tab) (read as a PDF, checked 26 Oct 2026)
• OpenTelemetry, sampling: head sampling is simple and efficient but cannot decide by the whole trace; tail sampling can always keep traces that contain an error, but it needs stateful systems that accept and store a large amount of data, and a tail sampler may need to fall back to simpler sampling if it cannot keep up. https://opentelemetry.io/docs/concepts/sampling/ (opens in a new tab) (checked 26 Oct 2026)
• Traffic, spans per trace, bytes per span, the price per GB, the error and slow shares, the bug rate, the 30-second wait and the engineer-weeks are our assumptions. The volumes, costs, 39.5%, 4.9%, 99.5% and 360 MB are our calculations, shown above.