Graceful shutdown: where the errors on every rollout come from

What is actually being asked

The ladder usually runs like this:

1. "What happens when a container is stopped?" — the warm-up, pointing back to the track's first lesson: SIGTERM, a grace period, SIGKILL.
2. "Why do rollouts produce errors?" — where the substance starts.
3. "What is a graceful shutdown?" — usually answered with "wait for the requests to finish", without saying which ones.
4. "In what order do you close things?" — the question that separates those who have done it.
5. "What about long requests that do not fit the grace period?" — about the limits of the method.
6. "And the load balancer?" — about the drain starting before the signal.

The numbers come from running bench/shutdown/drain.py and bench/shutdown/practice.py: a real server on loopback, twenty concurrent clients, 300 ms of work each, and a signal arriving halfway through it.

Base: a shutdown is a sequence, not an instant

Before talking about signals and draining it is worth naming, in ordinary words, what "stopping a service" even means — because every mistake in this topic comes from treating a shutdown as a single action.

A shutdown done properly is four actions in order:

1. stop taking new work — new requests are no longer picked up, they go somewhere that will serve them;
2. finish the work already in hand — the requests already being processed are carried through to a response;
3. close what was being used — database connections, files, connections to neighbouring services;
4. exit.

A process without that sequence performs only the fourth action. And here is the question this lesson is about: what happens to the work in hand if you jump straight to step four?

The answer is simple and unpleasant: nothing happens to it — it is simply gone. There is no process, so there is nobody to answer: a client that has already sent its request and is waiting gets a broken connection instead of a response. The request may have been half-done, and the client will never know.

That is where "a few 500s on every rollout" comes from. It is not a property of rollouts: it is what the process did with its own work at the moment it was asked to leave.

That is already enough to answer the basic interview question. Everything below is about how a process is told to stop at all, why the order of the first two actions matters more than it looks, and why all this politeness is given a finite amount of time.

Mechanism 1: what exactly gets broken

The first mode is what a process with no handler at all does (the track's first lesson), or one whose handler says "exit now":

1. EXIT ON SIGTERM, THE WAY A PROCESS WITHOUT A HANDLER DOES
------------------------------------------------------------
  requests broken                              20 of 20
  time from SIGTERM to exit, ms                2
  wait status: killed by signal                15
  the same as a shell reports it               143

Twenty out of twenty. Not "a few 500s" but every request that was in flight at that moment. Their share of the total flow is small only because the moment is short: what breaks is not a percentage of requests but whoever happened to be inside the second of the shutdown.

Hence the first clarification worth making in conversation: "0.1 % errors per deploy" is not a reliability figure but a product of processing time and deploy frequency. The same code with requests twice as long produces twice the errors without changing anything in itself.

Mechanism 2: draining is two actions, not one

The second mode differs by its handler. It does two things in order:

1. closes the listening socket — no new connections are accepted;
2. waits for the work on the connections already taken to finish, and only then exits.

2. STOP ACCEPTING, FINISH WHAT IS STARTED, THEN EXIT
----------------------------------------------------
  requests broken                              0 of 20
  time from SIGTERM to exit, ms                164
  exit code                                    0

Zero out of twenty. And note the price: 164 ms against 2 — the remainder of the work that was already running. The signal arrived halfway through 300 ms of processing, so about half of it had to be waited out.

The boundary of that zero is worth naming here, or a wrong rule comes out of it. Zero broken is the result of this run: this server, this load, 300 ms of processing, and a grace period the work in flight fitted into with room to spare. "Draining breaks nothing" is not a property of draining but a description of the case where the work finished in time. The rule is written from the boundary: a drain carries through only what fits into the time it is given, which is why the drain itself must be bounded in time. The reason is just below.

The order of the two actions matters, and that is the substance of the interview question. Wait first and close the socket afterwards, and new connections arrive during the wait — the drain never ends while the load lasts. Close the socket and exit immediately, and the same requests break as in the first mode.

Why this is possible at all is known from the first lesson: SIGTERM can be caught. The other half of the same rule is the method's boundary:

So a drain works only inside the grace period. Whatever has not finished will be cut off, and no handler changes that. Hence the cost of waiting indefinitely: a handler that waits "as long as it takes" does not get more time, it gets SIGKILL at the end of the period — and then everything still in flight breaks. The zero broken requests rest not on the handler's patience but on the work having finished before that moment.

Mechanism 3: closing the socket is not the same as refusing

A subtlety worth knowing: closing the listening socket stops new connections from being accepted, but connections already sitting in the kernel's queue do not disappear. From the previous lesson:

Connections pile up in the kernel's queue, not the application's. So at the moment of shutdown three different groups of requests exist, and each is handled differently:

• in flight — the drain finishes them;
• in the accept queue but not yet taken — closing the listening socket discards them; the client sees a reset although the application never knew they existed;
• not yet arrived — they go to another replica, if the balancer already knows this one is leaving.

Hence a third action that has nothing to do with the process itself: take the replica out of rotation first, and only then stop it. Draining inside the process does not help against requests still being directed at it.

Deeper: work that does not fit the grace period

A drain is bounded by the grace period, so there is work that will not fit into it: long exports, streaming responses, large transactions. The method does not save those, and it is worth saying so yourself.

Three ways, each with its own price.

Allow the interruption. If the operation is idempotent and the client retries (the lesson on retries), a break costs one retry. This is the cheapest option and the most common one in practice.

Make it resumable. Answer in parts with a position marker, and the client continues from where it broke off. More expensive in code, cheaper in operation.

Move it out of the request. The long work goes into a background job and the request only enqueues it. Then stopping the process breaks nothing important — and restarting the job becomes a separate mechanism with properties of its own.

How to answer in an interview

Short answer: a graceful shutdown is two actions in order — stop accepting new connections and finish the ones already taken — and it works only because SIGTERM can be caught. Measured on one and the same server under identical load: without a handler 20 requests out of 20 were broken, with draining zero, and the cost of draining was 164 ms against 2 — the remainder of the work already started.

That is enough for a correct answer. What follows is what you add when the interviewer digs.

If the interviewer digs deeper

Three things separate a good answer. First, you name the order: the socket closes first, otherwise the drain never ends under load. Second, you talk about the balancer: a replica is taken out of rotation before the signal, because draining does not cancel requests still being routed to it. Third, you know the boundary: draining lives inside the grace period, and SIGKILL cannot be caught, so long work must be made resumable or moved out of the request.

And one thing that is easy to overdo. Saying "with draining nothing ever breaks" passes a single run off as a property of the method. Zero out of twenty came out on one particular server, under one particular load, with work that fitted inside the grace period. The precise phrasing is this: a drain carries through exactly what fits into the time it is given — which is why the drain is bounded in time itself, or SIGKILL ends it and the breakage comes back.

Next they ask

Common misconceptions

Practice

Two exercises. Answer first, then check against the real output: in both, the correct answer comes from a script's committed output rather than being written by hand.

abrupt = run("abrupt")
drained = run("drain")
print(abrupt["broken"])
print(drained["broken"])
print(abrupt["exit"])

20
0
-15

20
0
143

1
0
-15

20
20
-15

Knowledge check