Retries and jitter: why a service falls over a second time at the moment it recovers

What is actually being asked

The ladder usually runs like this:

1. "What do you do when a request fails?" — the warm-up, where the word "retry" appears.
2. "How many times and with what pause?" — where the substance starts.
3. "Why exponential backoff rather than a fixed pause?" — a question about load amplification.
4. "What is jitter and what is it for?" — the central question.
5. "The service was down, came back and fell over again immediately — why?" — the thundering herd.
6. "How do you bound retries at the system level?" — a question about retry budgets.

The numbers in this lesson come from a model rather than from a run against a real system. The thundering herd is a property of the retry schedule, and seeing it takes a thousand clients and a reproducible outage. The model gives both; the price is that every conclusion holds exactly as far as the assumptions do, which is why they are named one by one.

Base: why retry at all, and why it is dangerous

Start with the simple part. A request to a service failed. The cause varies, and one of the possible causes is temporary: the service happened to be busy, overloaded, or restarting at that moment. A second later the very same request will go through. So the first thing that comes to mind is to retry — and the thought is a sound one: a retry often does help.

Now add to the picture what is invisible from one client's seat. The service does not have one client. It has a thousand, and the code inside them is the same. When the service stops answering, it is not one request that fails but all of them — and all of them are retried at once.

Two unpleasant things follow, and neither is visible while you look at a single client.

The first: a retry adds load. While the service is not answering it receives not fewer requests than usual but more: to every client's first attempt a second, a third and so on get added. That extra traffic arrives exactly when it is hardest to take.

The second: retries arrive together. If every client waits the same amount between attempts — and they do, because the same code picks the pause — then everyone's next attempt lands on the same instant. A service that has just come up receives a wave rather than a steady stream.

And here is the question this lesson is about: how do you retry so that the retry helps rather than finishes the service off? The answer has two parts, and they answer different questions. Backoff — how much the pause grows after each failure — governs how many retries go into a service that is not answering. Jitter — randomness in choosing the pause — governs when they arrive. Neither substitutes for the other.

That is already enough to answer the basic interview question. Everything below is about how large both effects are in the model and why the tidiest-looking strategy turns out to be the worst.

Mechanism 1: a constant interval is not "the simple option"

The first strategy anyone writes without thinking: retry every hundred milliseconds until it works.

CONSTANT INTERVAL
-----------------
  first wave at t=0 (all clients at once)  1000
  peak in one 10 ms step after recovery    0
  when that peak happened, ms              0
  requests sent into the outage (wasted)   20000
  total requests sent                      20000
  clients served                           0
  all clients served by, ms                -1

Two numbers should be read together. 20,000 requests went into a service that could not answer: a thousand clients spent twenty attempts each. And zero served: by the time the service came back, the clients had run out of attempts.

And a third line that is easy to read backwards: the peak after recovery is zero. That is not protection from a peak but its absence for a worse reason: by the time there was somewhere to come back to, nobody was left to come.

That is amplification: during a failure the load on the service that cannot answer does not drop but grows twentyfold — from the thousand first requests to twenty thousand — and it grows exactly when the service is at its worst.

Mechanism 2: exponential backoff is about rate, not about patience

The second strategy doubles the pause after each failure: 100 ms, 200, 400, 800. This is not an invention of application libraries — it is how the kernel itself behaves when retransmitting the packet that opens a connection:

The doubling is proved in that line by arithmetic: six retries add up to 127 seconds in exactly one case — pauses of 1, 2, 4, 8, 16, 32 and 64. With a constant pause, six retries would fit into a few seconds.

The point of doubling is reducing the rate, not being polite. Each further failure means the previous estimate of the situation was too optimistic, and the right response is to ask less often.

In the model it looks like this:

EXPONENTIAL, NO JITTER
----------------------
  first wave at t=0 (all clients at once)  1000
  peak in one 10 ms step after recovery    1000
  when that peak happened, ms              3100
  requests sent into the outage (wasted)   5000
  total requests sent                      6000
  clients served                           1000
  all clients served by, ms                3100

5000 requests wasted instead of 20,000 — four times fewer. And every client was eventually served.

But look at the line the lesson exists for: the peak after recovery is 1000 requests in a single ten-millisecond step. Everyone waited the same, so everyone came back at the same time. A service that has only just got up receives the whole thousand at once.

Mechanism 3: jitter is about the schedule, not about politeness

The third strategy differs by one line: the pause is chosen at random from zero up to the exponential bound.

EXPONENTIAL + FULL JITTER
-------------------------
  first wave at t=0 (all clients at once)  1000
  peak in one 10 ms step after recovery    10
  when that peak happened, ms              2000
  requests sent into the outage (wasted)   5906
  total requests sent                      6906
  clients served                           1000
  all clients served by, ms                13290

The peak fell from a thousand to ten — a hundredfold. No client backed off: each still retries until it succeeds. What changed is that they stopped doing it at the same time.

Hence the phrasing worth answering with: jitter does not reduce the load, it spreads it. The load is created by the retry itself — jitter only decides at what moment the retries arrive. The total number of requests even grew slightly because of it (6906 against 6000), because random pauses are sometimes shorter than regular ones.

And here is the price usually forgotten in conversation: everyone is served by 13,290 ms instead of 3100. Having spread the peak, we spread the recovery. It is a real trade-off rather than a free improvement: jitter is chosen when the peak is more dangerous than the delay — which is almost always true for a service that falls over under a peak, and not always true for a client that needs an answer soon.

And straight away the boundary, without which that number is dangerous to carry off. 13,290 ms is not "the price of jitter" in general. It is the result of this model under these assumptions: a thousand clients, an outage of exactly two seconds, exponential bounds of 100, 200, 400 and 800 ms, and the random seed 20260905. Change the seed and the number changes; change the length of the outage or the number of clients and it changes too. What reproduces here is the direction: the recovery stretches out, and it stretches further the wider the range the pause is drawn from. So the rule is written from the direction rather than from the number: a peak that has been cut is always paid for by a later "everyone served", and that price is measured in your own system rather than taken from here.

Deeper: what the model does not show

Three things worth saying yourself — and that is part of a good answer too.

Retries are not free for the client making them. Every retry holds a connection and a worker on the client side; with mass retries the client hits its own limits (earlier lessons: descriptors, the accept queue) before the service manages to come back.

Not everything may be retried. A retry is safe for an idempotent operation: a read, a write under a key, a cancellation by identifier. For "take the money" a retry is a second withdrawal, and what is needed there is not a pause but an idempotency key.

A retry budget bounds the system, not the client. A rule like "the share of retries in the total flow has a ceiling" is exactly what the model does not contain: it simulates the behaviour of one layer, while the budget lives at the level of the whole system. Without it every layer of a chain multiplies its neighbour's retries, and three layers with three attempts each turn one request into twenty-seven.

How to answer in an interview

Short answer: retrying amplifies load at the moment of failure, and an identical retry schedule synchronises that load into a peak after recovery. So two different things are needed: exponential backoff, so that there are fewer retries as failures repeat, and jitter, so that clients do not come back together. In this lesson's model jitter cut the peak a hundredfold but pushed the moment everyone is served from 3100 ms out to 13,290.

That is enough for a correct answer. What follows is what you add when the interviewer digs.

If the interviewer digs deeper

Three things separate a good answer. First, you keep two effects apart instead of conflating them: the amount of load is created by the retry itself and reduced by backoff (5000 requests instead of 20,000 in the model), while the shape of the load is set by the schedule and changed by jitter, which takes nothing away (6906 requests against 6000, and a peak of 10 instead of 1000). Second, you mention idempotency: not every operation may be retried, and a pause does not help there. Third, you name the retry budget as a system-level bound, because without it every layer multiplies the previous one's retries.

And one piece of care with the numbers. The price of jitter — "everyone served by 13,290 ms instead of 3100" — is worth quoting, but with the caveat that it is the result of one model under one random seed rather than a constant. The only universal part is the direction: a peak that has been cut is paid for by a later moment at which everyone has been served. Saying 13,290 without that caveat passes one run off as a law.

Next they ask

Common misconceptions

Practice

Two exercises. Answer first, then check against the real output: in both, the correct answer comes from a run of the model rather than being written by hand.

constant = simulate(delay_constant)
plain = simulate(delay_exponential)
jittered = simulate(delay_full_jitter)
print(plain["peak_after"])
print(jittered["peak_after"])
print(constant["served"])

1000
10
0

1000
1000
1000

10
1000
0

1000
10
1000

Knowledge check