Deep Engineering
Intermediate·Published·25 MIN

SLOs and error budgets: why "99.9 %" without a window means nothing

The same target means 43 minutes a month and a minute and a half a day. The lesson works through the arithmetic of the window, shows that an invisible incident eats more budget than a loud one, and explains why choosing between a calendar and a rolling window is choosing the rule by which a deployment freeze ends.

Full technical treatment

TL;DR

Reliability is counted as a fraction, and promised as a level of that fraction over a named period. Out of a thousand requests, 990 successful is 99 % availability. The promise "we hold 99 %" sets the level, and everything below the level is how much failure the promise allows: ten requests in a thousand. Until the period the fraction is counted over is named, such a promise can be read any way you like.

Hence the main consequence: the same number means different things, and the most expensive incident is the one nobody noticed. The arithmetic: 99.9 % is 43.2 minutes over 30 days and 1.4 minutes over a day. And the price of incidents ranks them differently from the feeling: a total outage of 10 minutes is 23 % of the monthly budget, while two percent of requests failing for a day — which nobody will see — is 67 %.

Beyond that are units, windows and boundaries. The unit of an error budget follows from the SLI: with availability counted over requests, the budget is conveniently counted in failed requests — at 1000 requests per second with a 99.9 % target over 30 days that is 2,592,000 requests; with availability counted over time, the budget is counted in time. Half the requests for an hour is 1,800,000 errors, or 69 % of the month's budget. The window is the rule by which a deployment freeze ends: after an incident on day 28 of a calendar month the budget returns on the 1st, while a rolling window returns it only 30 days after the incident itself. And all of this is arithmetic and a model rather than a measurement: the numbers follow from the target, the window and the incident profile under explicitly stated assumptions.

Where to start
Before this lesson it is enough to understand
  • a request to a service can end in success or in an error, and errors always happen;
  • a fraction is division: how many succeeded out of how many in total;
  • rolling out a new version is a risk: something may break because of it.
You do not need to know in advance
  • SLI, SLO, SLA and the error budget — all four words are taken apart here, from the start;
  • observation windows, calendar against rolling, budget burn.

What is actually being asked

The ladder usually runs like this:

  1. "What are SLI, SLO and SLA?" — the terminology warm-up.
  2. "What does 99.9 % mean?" — where it becomes clear whether the candidate asks about the window.
  3. "How does an error budget follow from an SLO?" — where the substance starts.
  4. "What do you do when the budget is spent?" — about a decision rather than a metric.
  5. "Calendar window or rolling window?" — the question that separates those who have used one.
  6. "Why not just set the target at 100 %?" — about the purpose of a budget.
model with assumptionsThe numbers here are arithmetic and a model from bench/slo/budget.py rather than a measurement of a live system. Assumptions: a constant 1000 requests per second, a rectangular incident, a budget counted as a share of failed requests.

This lesson rests on calculation rather than on a run. The reason is simple: an error budget is arithmetic over two numbers, and everything in it is derived rather than measured. The script prints both the inputs and the result, so every number here can be recomputed.

Base: four words that usually get mixed up

Start with arithmetic on your fingers, without a single term.

A service received 1000 requests. 990 succeeded and ten returned an error. So availability over that thousand is 99 %, and the amount of "bad" that happened is ten requests out of a thousand. That is all the mathematics there is in this topic: everything else is agreements about what to count, what level to want, and what to do when the level is not held.

Now those four words, each in ordinary language.

SLI — what we measure. The quantity itself: the share of successful requests. It could be another one: the share of requests faster than 300 ms, the share of completed orders. Until a quantity is chosen there is nothing to argue about.

SLO — the level we want. A target on the chosen quantity: "the share of successful requests is not below 99 %". An internal decision of the team, not a promise to the outside world.

The error budget — how much failure we allow ourselves. The same number turned the other way round: if the target is 99 %, then ten failures per thousand requests are permitted. Not "ten errors are bad" but "ten errors are paid for in advance".

SLA — the external obligation. What is written into a contract with a customer, with consequences for breaking it. It lives separately from the SLO and is usually looser: the internal target is held tighter than what is promised outside, so that the gap between them stays a place for engineering decisions rather than for lawyers.

And here is the question this lesson is about: which thousand? The 990-out-of-1000 fraction was counted over one thousand requests. An hour holds one number of them, a month an entirely different one, and the same target over those stretches means completely different things.

That is already enough to answer the basic interview question. Everything below is about what a target becomes once a period is named, what unit a budget is counted in, and what happens when it runs out.

Mechanism 1: a target without a window has no meaning

1. THE SAME TARGET, DIFFERENT WINDOWS
-------------------------------------
    target          1d          7d         30d         90d
    99.00%    14.4 min       1.7 h       7.2 h      21.6 h
    99.50%     7.2 min    50.4 min       3.6 h      10.8 h
    99.90%     1.4 min    10.1 min    43.2 min       2.2 h
    99.99%         9 s     1.0 min     4.3 min    13.0 min
model with assumptionsbench/slo/budget.py. Pure arithmetic: the allowed share of failure multiplied by the length of the window.

Read the 99.90 % row left to right: 1.4 minutes over a day, 10.1 minutes over a week, 43.2 minutes over a month, 2.2 hours over a quarter. One promise, four different quantities.

Hence the first thing to establish in conversation: "we run three nines" is not yet a promise. Three nines a day is a hard target where one restart with a minute's pause eats most of the day's budget. Three nines a quarter is a target that permits a two-hour outage once in three months.

Hence, too, the practical consequence for negotiation: with no window named, the two sides understand the promise differently and discover it during the first incident.

Mechanism 2: the unit of a budget follows from the SLI

An error budget is "how much failure is allowed". The first question here is failure measured in what, and the answer is not a matter of taste: the unit of the budget follows from the SLI.

  • The SLI is counted over requests ("the share of successful requests") — the budget is conveniently counted in failed requests.
  • The SLI is counted over time ("the share of minutes the service was up") — the budget is counted in time, in minutes of unavailability.

In this lesson the SLI is the share of successful requests, so everything from here on is counted in requests, with minutes standing beside them as a conversion. That conversion is exact only for a total outage: there every dead minute is all of that minute's requests, and only for that reason can 2,592,000 requests be called "43.2 minutes".

2. WHAT ONE INCIDENT COSTS
--------------------------
  target                                       99.9%
  window, days                                 30
  requests in the window                       2 592 000 000
  error budget, requests                       2 592 000
  error budget, minutes of total outage        43.2 min

  incident: 100% of requests fail for 10 min   600 000 requests = 23% of the budget
  incident: 50% of requests fail for 60 min    1 800 000 requests = 69% of the budget
  incident: 2% of requests fail for 1440 min   1 728 000 requests = 67% of the budget
model with assumptionsbench/slo/budget.py. Traffic is taken as constant at 1000 requests per second; with uneven load the same percentages give different absolute numbers.

Three incidents that feel completely different — and a price that ranks them differently from the feeling.

A total outage of ten minutes is what everyone notices, what reaches the internal chat and what gets a write-up. It costs 23 % of the monthly budget.

Two percent of requests failing for a day is what nobody notices: the charts are nearly flat, a few users retried and forgot. It costs 67 % — three times the loud outage.

Hence the lesson's central idea: a budget measures volume, not loudness. It exists precisely to put on equal footing two kinds of damage that intuition rates incomparably.

And note what makes that comparison possible at all: both incidents are counted in one unit, in failed requests, because that is what the SLI is here. Were availability counted over time, there would be nothing to measure the quiet degradation with — the service answered all day and was never down for a single minute.

And hence the practice: quiet degradation is more dangerous than noisy failure, because nobody hurries to fix it. What discovers it is the budget count — not an eye and not an on-call engineer.

Mechanism 3: calendar window against rolling window

The third block is about what happens after an incident.

3. CALENDAR WINDOW AGAINST ROLLING WINDOW
-----------------------------------------
  budget for 30 days, requests                 2 592 000
  incident on day 28: 50% for 1 hour           1 800 000 requests
  that is                                      69% of the budget

  calendar window: the budget resets on the 1st
  budget left on day 29                        31%
  budget left on day 31 (new month)            100%

  rolling 30-day window: the incident stays for 30 days
  budget left 1 day after the incident         31%
  budget left 15 days after the incident       31%
  budget left 29 days after the incident       31%
  budget left 31 days after the incident       100%
model with assumptionsbench/slo/budget.py. One incident, two accounting policies; the difference shows not at the moment of the failure but several days later. The return of the budget to 100 % in either policy is set by the window rule rather than computed: the block shows what that rule looks like day by day.

One incident. The difference is when shipping may resume.

A calendar window forgives on a schedule: an outage on the 28th costs two days of caution, and on the 1st the budget is full again. That is convenient for reporting and creates a familiar skew: the end of the month is the time to take risks, the beginning is the time to ship nothing important, because "the budget has only just started".

A rolling window forgives nothing by calendar: an incident counts for exactly 30 days and then falls out of the window on its own. It describes what the user lived through more honestly, but it requires explaining why shipping is forbidden today when it was allowed yesterday and nothing has happened.

Both policies are legitimate, and the interview question is usually not "which is correct" but "do you see that the choice is not about a metric but about the rule that ends the freeze".

Deeper: what a budget is for, and why targets sit below 100 %

The last level is about the purpose of the construction and its limits rather than its arithmetic.

An error budget turns reliability from an argument into arithmetic with a shared answer. While budget remains, shipping is allowed: the risk is paid for in advance. When it is spent, deployments stop — not as punishment but because further risk has nothing to cover it.

Hence the answer about 100 %. A hundred-percent target means a zero budget, that is, a ban on any change: every release is a risk, and there is nothing to pay with. That is why targets are set below one deliberately — not because nobody can do better.

And hence, too, why a budget is counted on a metric the user sees. A metric like "the service answers its health check" (the next lesson) stays perfectly happy while users receive errors; a budget counted on it never runs out and stops nothing.

How to answer in an interview

Short answer: an SLO is a target on a metric plus a window, and an error budget is how much failure that target allows within that window — counted in the same unit as the metric itself. With availability counted over requests, 99.9 % over 30 days at 1000 requests per second is 2,592,000 failed requests, which for a total outage equals 43.2 minutes; the same 99.9 % over a day is 1.4 minutes.

That is enough for a correct answer. What follows is what you add when the interviewer digs.

If the interviewer digs deeper

Three things separate a good answer. First, you ask about the window immediately, because without it the target is not a number. Second, you know that quiet degradation costs more than a loud outage: two percent of errors for a day is 67 % of the monthly budget against 23 % for a ten-minute total outage. Third, you say that a budget exists for the sake of a decision: while it lasts, deployments go ahead; when it is gone, they stop — and that is the whole point of a target below a hundred percent.

And one thing that is easy to overdo. Saying "a budget is counted in requests, not in minutes" passes one particular case off as a rule. The unit of the budget follows from the SLI: availability counted over requests gives a budget in failed requests; availability counted over time gives a budget in time. The mistake is not minutes as such but the conversion between the two: "43.2 minutes" holds exactly as far as the incident resembles a total outage — which is also why quiet degradation does not register in minutes at all, since the service was never down for one.

Next they ask

Next they ask

How does an SLO differ from an SLA?

Short answer

An SLO is an internal target used for decisions about shipping; an SLA is an external obligation with consequences in a contract. In practice the internal target is usually stricter than the external obligation: the gap between them is where decisions live without legal consequences.

From which the answer to "what if we break the SLO" follows: nothing legal happens, but deployments stop. If an SLA is broken, the conversation turns to compensation, and that is no longer an engineering decision.

Next they ask

Which metric should a budget be counted on?

Short answer

An SLI is the metric itself: the share of successful requests, the share of requests faster than a threshold, the share of completed operations. An SLO is a target on that metric plus the window it is counted over, and the budget is counted in the unit of that same metric: over requests means failed requests, over time means time. A budget must be counted on an SLI that describes the user's experience, and that is why an SLI definition includes a threshold: "successful" and "faster than 300 ms" are different metrics with different budgets.

A metric like "the service answers its health check" is unfit for a budget: it can stay green while users receive errors, and a budget counted on it never runs out.

Next they ask

What do you do when the budget is spent?

Short answer

Stop the deployments that spend it and work on causes. The point is not punishment: changes are the main source of risk, and when there is nothing to pay for risk with, it is sensible to stop creating it.

It helps to agree in advance on what exactly stops: product deployments yes; fixes that reduce the spend, no. Otherwise the rule turns into a ban on repairs.

Next they ask

Why not set the target at 100 %?

Short answer

Because that is a zero budget and therefore a ban on any change: every deployment creates risk and there is nothing to cover it with. A hundred-percent target does not mean "we are very reliable" but "we have no right to change anything".

There is a second reason: a hundred percent cannot be measured. Every metric has its own gaps and collection failures, and a target equal to the limit breaks on the first failure of the measuring system.

Common misconceptions

Claim

99.9 % is a clear target

Actually

Without a window it is not: the arithmetic gives 1.4 minutes a day and 43.2 minutes over 30 days from the same number. Until the window is named, the two sides understand the promise differently and find out during the first incident.

Claim

a loud outage always costs more than quiet degradation

Actually

More often it costs less: a total outage of 10 minutes is 23 % of the monthly budget, while two percent of errors for a day is 67 %. A budget measures the volume of damage rather than its loudness, and quiet degradation is the more dangerous precisely because nobody hurries to fix it.

Claim

an error budget is always allowed downtime

Actually

The unit follows from the SLI. Count availability over time and the budget really is downtime. Count it over requests, as this lesson does, and the budget is a number of failed requests: at 1000 requests per second with a 99.9 % target over 30 days, 2,592,000 of them. The mistake is not minutes but the conversion: "43.2 minutes" is exact only for a total outage, and two percent of errors for a day does not show up in downtime at all.

Claim

calendar and rolling windows are a matter of taste

Actually

They are different rules for ending a freeze. After an incident on day 28 a calendar window returns the budget on the 1st, a rolling one only 30 days after the failure itself. One incident, two different answers to "may we ship today".

Claim

a 100 % target is simply a very high bar

Actually

It is a zero budget, that is, a ban on any change: every deployment creates risk with nothing to cover it. And a hundred percent cannot be measured — every metric has gaps of its own, and a target equal to the limit breaks on the first collection failure.

Claim

a budget can be counted on the service's availability to health checks

Actually

Then it never runs out. A health check answers even while users are receiving errors (the next lesson), so budgets are counted on a metric that describes the user's experience: the share of successful requests or of requests faster than a threshold.

Claim

a spent budget means stopping all work

Actually

What stops is what spends the budget — product deployments. Fixes that reduce the spend never stop, or the rule turns into a ban on repairs. That is worth agreeing on in advance rather than during an outage.

Practice

Two exercises. Answer first, then check against the real output: in both, the correct answer comes from a run of the calculation rather than being written by hand.

Practice · predict the output

A 99.9 % target with 1000 requests per second. Three things are printed: the 30-day budget expressed as total outage, the same budget over a single day, and what percentage of the monthly budget a ten-minute total outage costs. What does this code print?
budget = budget_requests(TARGET, WINDOW_DAYS)
print(human_minutes(WINDOW_DAYS * 24 * 60 * (1 - TARGET)))
print(human_minutes(1 * 24 * 60 * (1 - TARGET)))
print(f"{incident_errors(1.0, 10) / budget * 100:.0f}")

Practice · estimate

A 99.9 % target over 30 days. For one day the service fails two percent of requests, quietly enough that nobody notices. What percentage of the monthly budget does that day consume?
%

Knowledge check

Question 1 of 6

What does a 99.9 % target mean with no window given?

Sources & further reading

1 SOURCE

  1. The calculation behind this lesson: the error-budget modelSource. This lesson has no external primary source, and that is stated plainly: everything in it is arithmetic derived from two numbers (a target and a window), plus an incident model with declared assumptions. The assumptions: a constant 1000 requests per second around the clock, a rectangular incident, a budget counted as a share of failed requests rather than as downtime, and a rolling window that moves a day at a time. The run prints both the inputs and the result, so the whole calculation can be recomputed./en/bench/slo/budget.py