Timeouts and deadlines: the client left, the work carried on

What is actually being asked

The ladder usually runs like this:

1. "Why do you need timeouts?" — the warm-up.
2. "What timeout would you set on a call to a neighbouring service?" — where it becomes clear whether the candidate separates a connection timeout from a response timeout.
3. "The client timed out and left. What happens on the server?" — where the substance starts.
4. "How does a deadline differ from a timeout?" — the central question.
5. "Every hop has a one-second timeout, yet the request takes three. Why?" — the question about summing.
6. "How do you cancel work that is no longer needed?" — answered differently in different languages, though the mechanism is one.

The numbers come from running bench/deadlines/chain.py and bench/deadlines/practice.py over loopback. The 400 ms delays are set by the script itself; what carries meaning here is not the milliseconds but the counters of work done and work refused.

Base: three different questions about time

Take the most ordinary chain there is. Service A calls service B, B calls service C, and only then does the answer travel back. The client knocking on A is prepared to wait no more than a second: any longer and it simply leaves.

Three different things get asked about time in that chain, and in conversation they are constantly substituted for one another:

• a timeout — how long to wait. It is a duration, and it belongs to whoever waits: A decides for itself how long it is prepared to wait for B.
• a deadline — until what moment. Not a duration but a point in time: "the answer is needed no later than such-and-such an instant", and there is one of those for the whole chain, however many hops it has.
• cancellation — a request to stop working. That one is not about time at all but about a message: somebody has to tell the worker that the result is no longer wanted.

And here is the question this lesson is about: the client left — what happens to the work it ordered?

The answer the rest of the lesson grows out of: an expired timeout is an event only for the party that was waiting. It stopped waiting, and that is all. B and C learned nothing about it and keep computing: no message went to them, because a timeout is not a message.

A deadline does not close that hole by itself. A deadline is merely a value in the request, and it can stop the callee only under two conditions. First, it has to arrive: the protocol between A and B must carry it, or B will never know about it. Second, the callee has to honour it — to look at the time left before each step and refuse when there plainly is not enough. Take either condition away and the deadline stays a note on the caller's side, behaving exactly like a timeout.

That is already enough to answer the basic interview question. Everything below is about what continuing work costs, what a deadline that really did arrive looks like, and why one-second timeouts at every hop do not give one second for the whole chain.

Mechanism 1: a timeout belongs to the one who waits

Start with what a timeout is at the socket level:

Note what the description does not contain: a single word about the other side. A timeout is the waiting party's decision to stop waiting. No message goes to the worker: it never learns that its result is no longer wanted.

Check it with a chain of three services. Each "works" for 400 ms, the client waits 500 ms:

1. THE CLIENT GAVE UP; THE WORK DID NOT
---------------------------------------
  client timeout, ms                             500
  services in the chain                          3
  work each service does, ms                     400
  what the client got                            client timeout
  how long the client waited, ms                 501
  services that did the work anyway              3
  services that refused before starting          0
  services never called at all                   0

The client left. All three finished. Not one refused — nobody had told them anything.

Hence a cost that usually goes uncounted: after the client leaves, the work keeps holding a connection and a worker — exactly what this run shows. In a real service the worker also holds everything it had already taken: a database connection, a slot in a pool, memory. Under load that is what turns a short spike into a sustained failure — the system is busy computing results nobody is waiting for, and has nothing left for new requests.

Mechanism 2: a deadline travels with the request

A deadline is not "a better timeout" but a different quantity. A timeout is a duration each party counts from itself. A deadline is a point in time, shared by the whole chain and passed along with the request.

The difference shows on the same chain. The only change: the client announces a budget, and each service subtracts what it spent and passes the remainder on:

2. THE SAME CHAIN WITH A DEADLINE PASSED ALONG
----------------------------------------------
  budget the client announced, ms                500
  what the client got                            deadline
  how long the client waited, ms                 401
  services that did the work anyway              1
  services that refused before starting          1
  services never called at all                   1

Three differences from the first block, all of them practical.

One service did the work instead of three. The first one made it (500 ms of budget against 400 ms of work), the second saw a hundred milliseconds left and refused without starting — and the third was never called at all; the run counts these separately: one refused before starting, one never called.

Note what did not happen: the client did not receive two thirds of a result. It received nothing — only a refusal. The saving is not a partial answer but the fact that two services out of three did not spend 400 ms each on work nobody was waiting for.

The client got an answer rather than silence. deadline is a meaningful refusal: it says the request did not fit the budget, rather than that the network disappeared.

The answer came earlier: 401 ms against 501. A refusal by deadline happens the moment it becomes clear there is no way to make it — not when somebody else's patience runs out.

And now the thing this block makes easy to miss: what worked was not the deadline but the cooperation. On its own a deadline stops nothing — it is a value, not a command. The second service refused without starting because two conditions held at once: the remaining budget arrived with the request, and the service looked at it before taking the work on. Remove the first — the protocol does not carry a budget — and the second service works its 400 ms just as in the previous block. Remove the second — the budget arrives and nobody reads it — and the outcome is the same. Which is why "we introduced deadlines" means nothing by itself: it means only that the budget is passed at every hop and that every callee is obliged to check it.

Mechanism 3: why per-hop timeouts do not add up

The most common practical puzzle: "we have one-second timeouts everywhere, where did three seconds come from?" The answer is that every hop counts time from itself.

3. WHY PER-HOP TIMEOUTS DO NOT ADD UP TO THE CLIENT'S
-----------------------------------------------------
  timeout each service sets for the next one, ms 1000
  services in the chain                          3
  what the client got                            done
  how long the whole chain took, ms              1203
  longest any single hop waited, ms              under 1000

Not one timeout was violated. Every service honestly waited less than a second. And the client waited 1203 ms, because the waits happen in sequence.

Hence the upper bound — but it must not pass for a measured one. The three seconds here are not measured, they are derived from how the chain is built: every hop gives the next one a 1000 ms timeout (a constant in the script), the waits do not overlap, so the total wait cannot exceed 3 x 1000 ms. The measurement gives 1203 ms — the chain fits inside the bound rather than reaching it.

Hence the rule worth answering with: a timeout bounds one wait, not a whole request. The bound on a whole request is called a deadline, and it is set once, at the entrance.

Hence, too, a way to review somebody's configuration: if a service sets outgoing timeouts equal to its own incoming one, the chain is already broken. An internal timeout must be smaller than the remaining budget, not equal to it.

Deeper: two different timeouts called by one word

The last level is about a trap in configuration. connect(2) documents a separate error:

That is the connection timeout — the time allowed for a connection to be established at all. It is about availability: the server is not answering the handshake, the queue is full (the previous lesson), packets are being lost.

The response timeout is a different thing entirely: the connection exists, the request is sent, and we are waiting for data. That one is about processing speed.

They get conflated constantly, and the price is this: a single five-second timeout "for everything" gives you a system that waits five seconds for an unreachable service and the same five seconds for a slow answer — while in the first case it is sensible to give up quickly and try another replica, and in the second it may well be worth waiting. The actual values depend on your network and your service — they do not follow from this measurement.

How to answer in an interview

Short answer: a timeout bounds the caller's waiting and says nothing to the worker, so the work continues after the client leaves; a deadline is a point in time that travels with the request, and a worker can refuse against it without starting. Measured on a chain of three services: without a budget all three finished the work, with a budget one did — the second refused without starting and the third was never called.

That is enough for a correct answer. What follows is what you add when the interviewer digs.

If the interviewer digs deeper

Three things separate a good answer. First, you say that per-hop timeouts do not add up to the client's: every hop counts from itself, and three one-second hops give three seconds — and you add that this is derived from how the chain is built rather than measured. Second, you name the cost of continuing work — a connection, a worker, and behind them everything the worker holds — and explain why under load that turns a spike into a failure. Third, you separate the connection timeout from the response timeout: they stand for different failures, and one shared number serves both of them badly.

And one qualification that shows you have actually deployed deadlines. A deadline does not stop the callee by itself: it stops it only where both conditions hold — the budget is passed at every hop, and every callee checks it before taking work on. A protocol that does not carry the budget, or a service that never looks at it, turns the deadline back into an ordinary caller-side timeout, with all the consequences of the first block.

Next they ask

Common misconceptions

Practice

Two exercises. Answer first, then check against the real output: in both, the correct answer comes from a script's committed output rather than being written by hand.

budget = int(CLIENT_TIMEOUT * 1000)
without_deadline, worked_without, _, _ = run(budget_ms=-1)
with_deadline, worked_with, refused, uncalled = run(budget_ms=budget)
print(without_deadline)
print(worked_without)
print(with_deadline)

client timeout
3
deadline

client timeout
0
deadline

client timeout
3
client timeout

deadline
1
deadline

Knowledge check