Rollouts and rollbacks: why a canary does not help without the right detector

What is actually being asked

The ladder usually runs like this:

1. "What rollout strategies are there?" — a warm-up in naming.
2. "Why is a canary better?" — where the substance starts.
3. "How do you tell that a canary is bad?" — the central question, and usually the skipped one.
4. "How long do you keep a canary?" — about sample size.
5. "How fast can you roll back?" — the question that matters more than the one about rolling out.
6. "What about database migrations?" — about what does not roll back.

This lesson rests on a model rather than on a measurement: rollout strategies are described by particular products' documentation, with no common standard between them. The model answers not "how much for you" but "what it depends on".

Base: how a running version gets replaced at all

The task sounds simple: requests are being served by a version that works, and they should be served by a new one. All of the difficulty is that you cannot switch one off and the other on unnoticed — between those two states somebody is serving a live stream of requests. There are usually four ways named for crossing that gap, and all four can be described without a single number.

Recreate. The old version is switched off, the new one switched on. Between them is an interruption during which the service does not exist. Honest and simple, and perfectly fine wherever the interruption has been agreed in advance.

Rolling replacement. Instances are replaced one at a time: one running the old version goes down, a new one comes up in its place. There is no interruption, but for a while both versions run at once and serve one shared stream.

Blue-green. A complete second set — the whole new version — is brought up alongside the old one, and when it is ready all the traffic is switched to it at once. The return works the same way: switch back.

Canary. A small share of the traffic goes to the new version while everything else keeps going to the old one. If nothing bad happens, the share is increased step by step to the whole.

Here is the thing worth taking away from that list: a canary on its own is not "a safe rollout", it is just a small sample of the new version. It decides nothing and protects against nothing; all it does is reduce the share of traffic that will meet the defect until somebody decides to stop.

And from there comes the lesson's main question: who decides to stop, and on what signal? Safety does not come from the strategy; it comes from there being a signal beside the strategy that the rollout is stopped by, and from that signal firing before the defect has done its damage. Without such a signal a small share merely stretches what is happening out over time.

The model below does not compare exactly those four but the ones actually chosen between in practice, and it labels them the way the runs do: blue-green, rolling, canary. Recreate is not in the model: a service with an agreed interruption needs no signal, because there is nothing to stop. A fourth participant is, feature-flag — a flag in the code: the same small share as a canary, but switched by the application itself as it handles a request rather than by the infrastructure.

That is already enough to answer the basic interview question. Everything below is about what happens when the signal exists but is badly chosen, and about why the time to return matters more than the time to roll out.

Mechanism 1: four strategies under a detector that counts errors

The first run: the alarm goes off when 200 errors have accumulated; then another minute to decide; then the rollback.

1. DETECTION BY ERROR COUNT: 200 ERRORS RAISE THE ALARM
-------------------------------------------------------
        strategy   affected   detected, s   rolled back, s
      blue-green        520            39              104
         rolling       1820           139              499
          canary        645           324              414
    feature-flag        495           606              666

The result is unexpected, and that is the model's main value: the canary (645) did worse than switching all traffic at once (520).

The mechanism: the canary keeps five percent of the traffic on the new version while the detector waits for two hundred errors — and on a small share it takes twenty times longer to collect them. The alarm fires at second 324, by which time the canary has already become a full rollout. Caution turned into delay.

Rolling is the worst result, 1820. The culprit is not the rollout but the rollback: ten replicas that were replaced one at a time come back one at a time. Between the alarm at second 139 and the completed rollback at second 499 nearly six minutes pass, and the defect runs on half the traffic or more throughout.

Mechanism 2: the same defect, a different detector

The second run differs in one thing: the alarm is raised not by the number of errors but by their rate — when the new version holds a bad share for thirty seconds.

2. DETECTION BY ERROR RATE: 30 SECONDS OF A BAD SHARE
-----------------------------------------------------
        strategy   affected   detected, s   rolled back, s
      blue-green        470            29               94
         rolling       1270            29              389
          canary         29            29              119
    feature-flag          4            29               89

Now the strategies behave the way they are described: the canary affects 29 requests, the flag 4, and switching at once 470.

Comparing the two runs is the lesson:

3. THE SAME STRATEGY UNDER THE TWO DETECTORS
--------------------------------------------
        strategy   by count    by rate   times fewer
      blue-green        520        470          1.1x
         rolling       1820       1270          1.4x
          canary        645         29         22.2x
    feature-flag        495          4        123.8x

The smaller the share of traffic on the new version, the greater the dependence on the detector. For switching at once, changing the detector barely matters (1.1x); for the flag, which starts at one percent, it matters 123.8x.

Hence the conclusion worth answering with: a rollout strategy and a detection method are one construction, not two. A canary without a detector able to see trouble in a small sample is not caution but postponement: the same defect, later and with no gain.

Mechanism 3: the rollback matters more than the rollout

Look at the "rolled back" column in the first run: 104, 499, 414, 666. That is the moment traffic has fully returned to the old version.

Rolling has the earliest detection of the three careful strategies (139 s) and the longest rollback: between the alarm and the end (499 s) lie five minutes of returning replicas. The flag has the latest detection (606 s) but ends at the moment of the decision — a minute after the alarm (606 → 666): returning is instant.

Hence a practical criterion to apply to your own system: count not "how we roll out" but "how long it takes to come back". The second determines the damage, because the defect lives from the start to the end of the rollback, not to the moment of detection.

Hence, too, why "how long to keep a canary" has both an upper and a lower bound. Too short and the detector never gets its sample, so you roll the defect further (the first run). Too long and you extend the time the system lives in two versions at once, which is a source of errors of its own: incompatible formats, diverging schemas, duplicated background jobs.

Deeper: what does not roll back

The last section is about the limits of the method — and it is worth raising before you are asked.

Database migrations do not roll back with the code. If the new version added a column and started writing to it, reverting the code deletes no data and restores no schema. So schema changes are split into compatible steps: first add and learn to read, then write, then remove the old — each step reversible on its own.

Sent messages and external calls cannot be recalled: the email is gone, the payment went through, the message is in the queue. A rollback restores behaviour, not consequences.

Warm caches and connections are cold after a rollback, as after a restart (the lesson on probes). So for the first minutes after a rollback the system performs worse than before the rollout — which has to be accounted for when the decision to roll back is made on a latency metric.

Hence a boundary worth saying out loud, because everything above this point took the rollback for granted: a rollback is not always available. If the new version has already changed the schema or the data itself, there is nowhere for the code to go back to: the old version cannot read what the new one wrote, and "put it back as it was" would be one more irreversible operation on top of the first. In that position what works is not a rollback but moving forward — a fix is shipped rather than the previous version, and it has to be shipped under the same outage, with no second attempt.

The difference is practical, and it has to be understood before an outage rather than during one. A rollback plan is valid exactly up to the operation after which it stops existing — which is why irreversible steps are separated from reversible ones, done one at a time, and deferred where possible. Then most rollouts keep a cheap way back, and the expensive way forward is left only where nothing else will do.

How to answer in an interview

Short answer: the damage from a bad rollout is determined not by the strategy but by the product of three things — the share of traffic on the new version, the time to detection and the time to roll back. So a canary only wins together with a detector able to see the problem in a small sample: in the model the same canary affected 645 requests with a detector waiting for 200 errors and 29 with a rate detector — twenty-two times apart.

That is enough to answer correctly. Beyond it is what you add when the interviewer digs.

If the interviewer digs deeper

Three things separate a good answer. First, you talk about the rollback rather than the rollout: in the model rolling detected the defect earlier than the canary but affected nearly three times more requests — 1820 against 645 — because returning ten replicas takes as long as replacing them. Second, you name the boundary: database migrations, sent emails and completed payments do not roll back with the code. Third, you connect canary size to detector sensitivity: too small a share means the alarm arrives late.

And one thing worth saying before you are asked: a rollback is not a universal cure but one of two options, and the second is sometimes the only one. Once a schema or data migration has been applied there is nowhere to go back to, and what works is no longer a rollback but moving forward with a fix. What follows from that is a way of working rather than an argument: irreversible steps are separated from reversible ones and taken one at a time, so that every rollout keeps a cheap way back for as long as possible.

Next they ask

Common misconceptions

Practice

Two exercises. Answer first, then check against the real output: in both, the correct answer comes from a run of the model rather than being written by hand.

canary_count = simulate("canary", "count")
canary_rate = simulate("canary", "rate")
rolling_rate = simulate("rolling", "rate")
print(canary_count["affected"])
print(canary_rate["affected"])
print(rolling_rate["affected"])

645
29
1270

29
645
1270

645
29
120

520
29
1270

Knowledge check