Release speed vs release safety

The decision

Every team must choose how a finished change reaches users. With deploy on merge, each merged change goes to all users at once, and the team fixes problems after. With a staged rollout, the change goes first to a small share of users, called a canary, and automatic checks decide whether it continues. With a release train, changes wait and ship together on a fixed day, after manual testing. Speed means customers get value sooner. Safety means fewer and shorter failures. The forces are customer impact, the error budget (the failure time your service goal allows), and how fast the team learns.

Choose deploy on merge when

• The product is early, has few users and has no strict service goal yet.
• A rollback is fast and safe, for example one click that restores the last version.
• Changes are small, hidden behind feature flags, and easy to switch off.

Choose a staged rollout when

• The error budget is tight, for example a 99.9% goal.
• You have enough traffic that 5% of users give a clear signal within minutes.
• You have good automatic checks: error rate, latency and key business events.
• A bad release costs money for every minute, as in payments or checkout.

Choose a release train when

• A change cannot be rolled back easily, such as a mobile app in a store or a database move.
• A regulator or customer needs manual sign-off or a fixed change window.
• You have little traffic, so a canary gives no clear signal.

The numbers that change the answer

All numbers here are our assumptions unless marked. The service goal is 99.9%, so the monthly budget is 30 x 24 x 60 x 0.001 = 43.2 minutes (calculation). A bad release fails all requests. The team ships 40 changes a month, and 15% of single merges fail. Detecting a bad release takes 8 minutes. A full rollback takes 7 minutes, so a bad release affects all users for 8 + 7 = 15 minutes. A canary rollback takes 5 minutes, so 8 + 5 = 13 minutes.

Deploy on merge (calculation).

• Bad releases a month = 40 x 0.15 = 6.
• Minutes of failure = 6 x 15 = 90. That is 90 / 43.2 = 208% of the budget.
• Cost: assumption of Rs 20,000 a minute of failure, so 90 x 20,000 = Rs 1,800,000 a month.

Staged rollout, 5% first (calculation). The Google SRE Workbook gives an example: a 5% canary with a 20% error rate gives a 1% error rate overall (see Sources). Our numbers use the same multiplication.

• Assumption: the canary catches 80% of bad releases. The other 20% reach everyone.
• Caught: 6 x 0.8 = 4.8 releases. Each hurts 5% of users for 13 minutes, which is 0.05 x 13 = 0.65 full-outage minutes. Total 4.8 x 0.65 = 3.12 minutes.
• Missed: 6 x 0.2 = 1.2 releases x 15 minutes = 18 minutes.
• Total 3.12 + 18 = 21.12 minutes (about 21), or 21.12 / 43.2 = 49% of the budget. Cost: 21.12 x 20,000 = Rs 422,400.
• Time to ship: Assumption: 4 steps of 10 minutes (5%, 25%, 50%, 100%) = 40 minutes. One rollout at a time gives at most 480 / 40 = 12 deploys in an 8-hour day.
• Cost to build: Assumption: 6 engineer-weeks x Rs 62,500 (an engineer at Rs 3,000,000 a year, over 48 weeks) = Rs 375,000.

Release train (assumptions plus calculation). Assumption: batches are bigger, so the change fail rate is 30%. A weekly train has about 4 releases a month. Bad releases = 4 x 0.3 = 1.2. Failure = 1.2 x 15 = 18 minutes, which is 42% of the budget, or Rs 360,000. A merged change waits 3.5 days on average (a week divided by 2).

The train and the canary burn about the same budget. The canary is far faster. Deploy on merge is fastest and breaks the goal unless rollback or flags make each bad release much shorter than 15 minutes.

Cited fact (DORA). DORA's research says speed and stability are not trade-offs. The metrics are correlated for most teams, and top performers do well on all five (see Sources).

What breaks with each choice

Deploy on merge.

• A defect hits everyone before anyone looks. Fallback: feature flags that switch a change off in seconds, and a one-click rollback.

Staged rollout.

• The canary does not see the defect, because it needs a rare user type or a rare time of day. Fallback: choose canary users from many segments, and keep a delayed alert for the full rollout.
• Two versions run together, so data written by one may break the other. Fallback: expand and contract changes, and flags for new data formats.
• The checks fail by mistake and block good releases. Fallback: a manual override with a log, and a review of every false alarm.
• The pipeline becomes the bottleneck (12 deploys a day). Fallback: batch tiny changes or run canaries for separate services in parallel.

Release train.

• Bigger batches hide the bad change: ten changes ship together and finding the cause takes hours. Fallback: flags, so you can switch one off.
• Engineers wait 3.5 days on average, and the context is gone when a bug appears. Fallback: a train twice a week.

What it costs

• People and time: about Rs 375,000 and 6 weeks to build a canary pipeline (Assumption), plus a part of one engineer to maintain the checks.
• Reversible-first path. Start with deploy on merge, one-click rollback and feature flags. Count bad releases and minutes of failure each month. Add a staged rollout when failure minutes pass about 30% of the budget (here 13 minutes) for two months (our trigger, an assumption). Use a train only for things that cannot be rolled back, such as a store app.
• Changing your mind: moving from merge to canary is a pipeline project of weeks. Moving back is easy. Moving from a train to continuous delivery is a change of habits, tests and trust, and takes quarters.

How each level answers

The question is: "How would you release a payments service 40 times a month?"

SDE-1: I test the change and deploy it. If it fails, I roll it back.

SDE-2: I use a canary: send 5% of traffic to the new version, watch errors, and then continue or roll back. I also use feature flags.

Senior: With a 99.9% goal we have 43.2 minutes a month. If 15% of 40 releases fail for 15 minutes each, we use 90 minutes. A 5% canary that catches 80% of defects cuts that to about 21 minutes. The fallback for the missed 20% is a flag and a fast rollback. The price is a 40-minute rollout.

Staff: I decide by what a failed minute costs us and what a waiting week costs the product. At Rs 20,000 a minute, the canary saves 1,800,000 - 422,400 = about Rs 1.4 million a month for Rs 375,000 of work, once. I give the platform team the pipeline, and each service team owns its checks. I measure the five DORA metrics, budget used and false alarms. I revisit it when the goal, the traffic or the team size changes.

Say it in the interview

I would deploy often behind feature flags and add a staged rollout. In my example, 40 releases at a 15% fail rate would use 90 minutes of a 43.2-minute budget. A staged rollout that starts at 5% lowers it to about 21 minutes if it catches 80% of bad releases. I accept a 40-minute rollout, and I keep a flag and a rollback for what the canary misses.

Common mistakes

• Saying "speed and safety are opposites". Small, flagged, staged changes help both. The cost is the work to build them.
• Trusting a canary with no traffic. At low traffic, 5% gives no signal. Use a longer bake time or another way to check.

Sources

• DORA, software delivery performance metrics: five metrics in two groups, throughput (three) and instability (two). The guide states that speed and stability are not trade-offs and that the metrics are correlated for most teams. https://dora.dev/guides/dora-metrics/ (opens in a new tab) (checked 26 Oct 2026)
• Google SRE Workbook, "Canarying Releases": a canary is a partial and time-limited deployment that is evaluated; the example of a 5% canary with a 20% error rate causing 1% overall impact. https://sre.google/workbook/canarying-releases/ (opens in a new tab) (checked 26 Oct 2026)
• The goal, fail rates (15% and 30%), detection and rollback times, the 80% catch rate, the cost of a minute, step times, the 30% trigger and the cost to build are our assumptions. The 43.2, 90, 21, 49% and 42% figures are our calculations, shown above.