Safe releases: canary, blue-green, rollback, feature flags

The interviewer asks: "You wrote a new version of checkout. How do you put it live so that a bug hurts very few users, and you can undo it in a minute?"

A weak answer is "I deploy at night and watch". A strong answer sends the new version to a small share of users first and compares it with the old one. It keeps a quick way back. It can also switch one feature off without new code on the servers.

At SDE-1 level you should name two ways to go live and one way back. At SDE-2 level you should choose one and say what to watch. At SDE-3 level you should also say what cannot be undone, such as stored data.

Think about a restaurant that adds a new dish. The chef serves it to one table and watches. If the table likes it, more tables get it. If not, the old menu is still ready.

Putting new code on your servers is a deploy. A release is the moment users start to get the new behaviour. Usually these are one act. Feature flags, below, can split them. Many outages start with a change (lesson 94). A safe release means that a bad change hurts few users and ends fast.

The blast radius of a change is the share of users that it can hurt. Here is a formula for the harm:

• failed requests per second = requests per second x share of traffic on the new version x failure rate of the new version

From a change to production: CI/CD in one picture

A pipeline is the list of steps that every change passes:

1. A developer pushes a change to the shared code store.
2. Build: a program turns the change into one package, called an artifact.
3. Test: automatic tests run on the artifact. A failed test stops the pipeline.
4. Staging: the artifact goes to a copy of the live system with no real users.
5. Production: the artifact goes live, in one of the safe ways below.
6. Watch: the pipeline or a person reads the metrics (lesson 91). If they turn bad, go back.

Steps 1 to 3 are continuous integration (CI): every change is built and tested automatically. Continuous delivery means that every passing change is ready to go live, and a person decides when. Continuous deployment means that a passing change goes live by itself. "CI/CD" names the whole pipeline.

Ways to go live

All at once. Replace the old version on every server together. The blast radius is every user, and the way back is another full release.

Rolling release. Replace servers in small batches, for example 2 of 20 at a time. A new server gets requests only after it passes its health check (lesson 28: a small test request on a timer). Plan for:

• Capacity. While a batch is out, the other servers carry its load.
• Two versions at once. Both versions use the same database for a while, so both must work with the same data (lessons 11 and 25).
• The start-up herd. A new server may load data from the database when it starts. A big batch does this all at once and can overload the database. Use small batches, and pass the health check only after loading ends.

A rolling release stops only when new servers fail the health check. A bug that passes the check reaches every server, unless someone watches.

Blue-green. You run two full copies of the live system. Blue serves the users. Green has the new version and no users. After a test, you change the load balancer (lesson 28) so that all requests go to green, in one step. To go back, you switch to blue again. This takes seconds, because blue still runs. You pay for double the servers while both run, and both copies share one database. Blue-green makes the way back fast. It does not make the first step smaller: after the switch, all users get the new version.

Canary. Send the new version a small share of traffic, such as 5%. The old version serves the rest and is the control. Compare the same numbers for both at the same time: errors and latency (lesson 91). If the canary is no worse, raise the share to 25%, 50% and 100%. If it is worse, send all traffic back to the old version.

Three choices make a canary work:

1. Who is in it. Choose users by a hash of the user ID. A hash is a fixed rule that turns an ID into a number, so one user always sees one version.
2. How long and how big. The canary needs enough requests to show a problem. A small problem needs more time. Include a busy hour.
3. The stop rule. Write it before the release, for example "go back if the canary fails 1 more request in every 100 than the control". A machine can check it.

The way back: rollback

Rollback means going back to the last good version. Keep the old artifact, so the pipeline can deploy it again. It is not always safe:

• If the new version changed stored data, the old version may not understand it. Expand first, contract last (lesson 25), so the old version keeps working.
• Emails that were sent and money that was moved do not come back.

When it is not safe, roll forward: ship a small fix as a new release. This is quick only if the pipeline is quick.

Feature flags

A feature flag is a setting, kept outside the code, that turns a feature on or off. It can change in seconds with no deploy. This flag turns on for a share of users:

import hashlib

def in_rollout(user_id, flag_name, percent):
    text = f"{flag_name}:{user_id}".encode()
    bucket = int(hashlib.sha256(text).hexdigest(), 16) % 100
    return bucket < percent

The hash gives each user a number from 0 to 99, always the same for that user. A user who is in at 5% is also in at 25%. Flags have three uses:

1. Dark launch. Deploy the code with the flag off. Now deploy and release are two separate acts.
2. Gradual rollout. Turn the flag on for staff, then for 1%, 5%, 25% and 100%.
3. Kill switch. A flag whose job is to turn a risky feature off during an incident.

Flags also cost something:

• Every flag makes two code paths. Test both, and remove the flag when the feature is fully on.
• The flag setting is a change too. A broken settings file sent to all servers at once can break all of them. Check the file before it ships, send it to a few servers first, and let a server keep the last good copy.
• If the flag store is down, the code needs a default. Choose the safe one, usually "off".

A/B tests in one paragraph

An A/B test uses the same split for another question: "do users do more of what we want with B than with A?". Users are split by hash into group A (old) and group B (new). You compare one business number, for example the share of users who order. Suppose 400 of 5,000 users order in group A and 430 of 5,000 in group B. This gap can easily be luck, and a statistical test says how likely it is. Choose the number, the group size and the end date before you start. A canary checks safety. An A/B test checks value.

We use the food-ordering app. The new checkout version has a coupon bug.

Assumptions (made up to teach with):

• Peak load is 1,000 requests per second on 20 servers, so each gets 50. One server handles 100 per second at full load.
• The new version fails 5% of its requests. The old version fails 0.1%.
• The team notices the problem 120 seconds after the bad version gets traffic.
• The way back takes 600 seconds for a full release, 60 seconds for the blue-green switch, and 60 seconds for canary traffic.
• A flag setting reaches all servers in 10 seconds.
• The harm stays the same until the way back ends (the worst case).

Step 1: all at once.

• Failed per second = 1,000 x 1 x 0.05 = 50. We count only failures of the new version.
• Time = 120 + 600 = 720 seconds. Failed requests = 50 x 720 = 36,000.

Step 2: blue-green. All traffic goes to green, so 50 requests fail each second for 120 + 60 = 180 seconds. Failed = 50 x 180 = 9,000.

Step 3: canary at 5%. One server of 20 is 5% of traffic.

• Failed per second = 1,000 x 0.05 x 0.05 = 2.5. This answers the predict box.
• Time = 120 + 60 = 180 seconds. Failed = 2.5 x 180 = 450. That is 36,000 / 450 = 80 times fewer than all at once.

Can the canary see the problem? It gets 50 x 60 = 3,000 requests a minute. At 5%, we expect 150 failures. The control at 0.1% gives 3 for the same number of requests. The gap is clear. A smaller bug that fails 0.2% gives 6 failures against 3, and luck hides the gap. In 30 minutes the canary gets 90,000 requests: 180 failures against 90, which is clear. So the stop rule must say how long to watch.

Step 4: feature flag. The coupon code is deployed with the flag off, so 0 requests fail. We turn the flag on for 5% of users. The code above puts about 5,000 of 100,000 made-up users in (a test run gave 5,047). Failures are 2.5 per second. The team notices after 120 seconds, and the kill switch needs 10 more: 2.5 x 130 = 325 failed.

Step 5: rolling batches. Each new server loads 200 MB from the database and needs 90 seconds before it is ready.

• Batches of 10: 10 servers are left, so each gets 1,000 / 10 = 100 per second, which is 100% busy. The database sees 10 x 200 = 2,000 MB at once. The release takes 2 x 90 = 180 seconds.
• Batches of 2: 18 servers are left, so each gets 1,000 / 18 = 55.6 per second, which is 56% busy. The database sees 2 x 200 = 400 MB at once. The release takes 10 x 90 = 900 seconds.

The small batch is slower but safer.

• A canary solves a small blast radius: 450 failures here, not 36,000. It costs a pipeline that compares two versions, a wait, and a stop rule. I would change to a plain rolling release when the service has little traffic, because a canary with few requests cannot show a problem.
• Blue-green solves a way back in seconds. It costs double the servers during the switch, and one shared database. I would add a canary when the bug risk is high, because all users get the bug at the switch.
• Feature flags solve a kill switch in seconds and a release apart from the deploy. They cost extra code paths and flags to remove. I would skip a flag for a tiny change that is easy to roll back.

"Blue-green makes a release safe." It makes the way back fast. After the switch, all users get the new version. With the same bug it lost 9,000 requests, and the canary lost 450.

"If something goes wrong, I will just roll back." Not always. If the new version changed stored data, the old version may fail on it. Sent emails and moved money stay as they are.

"The canary error rate looks the same as the control, so the release is good." A bug that fails 0.2% gives 6 errors against 3 in one minute. Luck can hide that gap.

1. Say in one sentence each: blue-green, canary and feature flag. In our example, which one gives the fastest way back? Why does blue-green not make the first step smaller?
2. A service gets 2,000 requests per second. A new version fails 3% of the requests it gets. A canary sends 10% of the traffic to it. The team notices after 90 seconds, and sending traffic back takes 30 seconds. (a) How many requests fail each second? (b) How many fail in total? (c) How many would fail if all traffic got the new version for the same time?
3. A release adds a new column and drops the old one in the same release. It goes out with blue-green. Three minutes after the switch, errors rise. A teammate says "switch back to blue". What can go wrong, and what do you do?