TIME_WAIT and ephemeral ports: the client hits the wall first

What this question is really about

The ladder looks like this:

1. "What is TIME_WAIT?" — the warm-up about the state after a close.
2. "Whose side does it appear on — the client's or the server's?" — the question that shows whether the rule about active close is known.
3. "We have 30 thousand sockets in TIME_WAIT, what do we do?" — the central question, and the one where answering with settings is visibly wrong.
4. "What does EADDRNOTAVAIL mean?" — a trap: it is not about memory and not about descriptors.
5. "How many new connections per second will a client take?" — a question about arithmetic rather than about "it depends".
6. "What is the state for in the first place?" — a question about what it protects.

The numbers come from running bench/timewait/ports.py and bench/timewait/practice.py over loopback. The duration of the state here is not taken from documentation but read off a clock.

Base: a connection leaves a trace behind after it closes

Before talking about states and port numbers it is worth naming, in ordinary words, what happens to a connection from start to finish. In short, its life goes like this:

1. the two sides connect — they agree that they are talking to each other;
2. the connection works — data travels over it in both directions;
3. it is closed — one side says "that is all from me" and the other answers in kind;
4. a waiting state — the conversation is over, but a trace of it is still held.

The whole lesson is about the fourth step. It has a name: TIME_WAIT.

Why that step exists. A network makes no promise that every packet arrived or that none of them was delayed on the way: a copy of a segment can wander across routes and turn up late. And a connection is identified not by a name but by four numbers — an address and a port on one side, an address and a port on the other. So if a connection is closed and a new one is opened straight away with the very same four, a late segment of the old conversation arrives into the new one — and is taken for its own, because there is nothing to tell them apart by.

The waiting state is the defence against exactly that: the side that closed does not hand its half of the four to anyone for a while. It waits until nothing of the old conversation can still be in the network.

That is already enough to answer the basic interview question: TIME_WAIT is not a leak and not a fault but the normal end of a connection. Beyond it comes which side gets the wait, how long it lasts, why a client runs out of port numbers because of it, and at what rate of connections that turns into a failure.

Mechanism 1: the wait goes to the side that closes actively

The first thing that settles half the questions in this topic: TIME_WAIT does not "appear in the system", it goes to a particular side of the connection. Which one is visible in a measurement where exactly one thing changes.

1. TIME_WAIT BELONGS TO WHOEVER CLOSED FIRST
--------------------------------------------
  the client closes first                        server side 0, client side 20
  the server closes first                        server side 20, client side 0

Twenty identical exchanges either way. The same server, the same client, the same data. The only difference is who called close first — and the sockets land entirely on that side.

Why that side and not the other. This is not an implementation quirk but a requirement of the standard:

The passive side has nothing to wait for: there is nothing left for it to acknowledge.

And here the rule is worth stating precisely, because the usual phrasing — "whoever closed first gets it" — is not always true. The standard is about active close, and on an ordinary close, where one side sends a FIN and the other answers with its own, that is the same thing as closing first: the measurement above was made exactly that way. But an ordinary close is not the only kind. There is a simultaneous close, where the two FINs cross on the way — then both sides close actively and both have to wait. And there is a reset, an abrupt tear-down instead of a FIN — a connection ended that way does not go through the waiting state at all. Neither is in this run: it shows the ordinary case. So the rule reads: the side performing the active close is the one that goes through TIME_WAIT, and "whoever closed first" is the special case of that for an ordinary close through FIN.

This is the practical point behind the question: thirty thousand TIME_WAIT on a server is not a complaint about clients but a fact about who closes connections. A server that answers and closes immediately accumulates them by construction. A server that keeps the connection open and lets the client close does not accumulate them at all.

The converse holds too: if the sockets pile up on the client, the client is the one closing actively — and the question is no longer "how do we remove them" but "why is there a new connection per request".

Mechanism 2: the duration is not tunable

The second mechanism is about what gets done instead of fixing the cause. How long the state actually lives:

2. HOW LONG IT LASTS
--------------------
  in TIME_WAIT right after close                 True
  seconds until it disappeared                   60.4
  tcp_fin_timeout on this machine                60
  tcp_max_tw_buckets on this machine             32768
  tcp_tw_reuse on this machine                   2

Sixty seconds on a clock. And that is where the trap is: tcp_fin_timeout sits right next to it at 60, and the resemblance looks like an explanation. It is not one — tcp(7) describes that setting differently:

Waiting for the other side's FIN is a different state and a different situation. It has nothing to do with the minute that was measured.

The two other settings people turn in this topic also do something else. On tcp_max_tw_buckets:

That is a bound on the count, not the duration — the quotation finishes the thought itself: If this number is exceeded, the socket is closed and a warning is printed. Such a bound does not solve "we are out of ports": it trades one problem for another.

The third setting, tcp_tw_reuse, is taken up below alongside SO_REUSEADDR: it is not about duration but about whether a port may be taken early. The value 2 in the run is not "enabled twice": on this machine reuse is permitted only for loopback, which is where the measurement runs.

The practical point: the right answer to "how do we shorten TIME_WAIT" is "you do not, and you do not need to". What is needed is different: stop creating that many connections, or stop being the side that closes actively.

Mechanism 3: what the client runs out of is port numbers

Now what the lesson exists for. What exactly runs out for a client that opens a connection per request:

3. WHAT THE CLIENT RUNS OUT OF IS PORT NUMBERS
----------------------------------------------
  port range for this block                      50000 50049, that is 50 numbers
  connections made                               50
  what stopped the loop                          EADDRNOTAVAIL

Fifty numbers, fifty connections. Not forty-eight and not fifty-two: exactly as many as there are numbers. There is nowhere for the fifty-first to come from, because every previous number is still held by that same waiting state.

And note the error itself. What it means is spelled out in connect(2):

Neither memory, nor descriptors, nor a refusal by the server appears in that definition: what ran out is numbers. The diagnosis follows — seeing this in a client's log, do not go looking on the server.

Mechanism 4: the ceiling is computed rather than guessed — but it is an estimate

Out of two measured quantities — the size of the range and the duration of the state — comes a number worth keeping in mind:

4. THE CEILING IS ARITHMETIC OVER THE TWO NUMBERS ABOVE
-------------------------------------------------------
  ip_local_port_range on this machine            32768 60999
  port numbers in it                             28232
  seconds a port stays in TIME_WAIT              60.4
  new connections per second, ceiling            467.7

About four hundred and seventy new connections per second — that is the whole headroom of an ordinary machine with default settings, if every request opens a new connection to one and the same destination.

The number is more modest than it sounds, and that is its value: a service at five thousand requests per second exceeds it tenfold, and no amount of machine capacity changes that — what runs out is numbers, not resources.

And straight away, how far the number reaches, because it is not a law of nature. It is a rough estimate for repeated connections to one and the same destination address and port in this configuration, not a ceiling of the machine. It has three boundaries. First: the kernel tells connections apart by four values — address and port on both sides — so a second destination gets its own set of numbers and the total is higher. Second: both quantities in the division were taken here, on this machine; with a different port range the number is different, and it has to be recomputed rather than remembered as a constant. Third: it is an estimate for the worst case — a client that reuses nothing.

That is precisely why the estimate is useful: not as a number in the machine's specification but as the order of load at which the habit of "a connection per request" stops working.

Mechanism 5: reuse removes the arithmetic entirely

The last mechanism is the shortest, because there is nothing to explain.

5. A REUSED CONNECTION SPENDS ONE PORT, WHATEVER THE LOAD
---------------------------------------------------------
  requests sent                                  400
  local ports spent                              1
  TIME_WAIT sockets left behind                  1

Four hundred requests, one port, one socket in TIME_WAIT at the end. The third block on the same machine failed at the fiftieth connection.

Same machine, same kernel. The third block ran with the range deliberately narrowed to fifty numbers — hence the failure at fifty. Reuse does not care about the size of the range: one port, whatever it is. One decision changed — keep the connection or close it. And that decision removes the ceiling arithmetic, the question about settings, and half the questions in the topic.

It also removes what the last two lessons were paying for: name resolution and the TLS handshake. One connection, one time each.

Deeper: the ports run out somewhere other than where the application runs

All the arithmetic above was computed for one machine: its own range of numbers, its own count of connections, its own error in its own log. In a real network that assumption often does not hold — and then the ports run out for someone other than whoever spent them.

Between the machines and the outside world there is usually a gateway that rewrites addresses: connections leave not with the address of the application's machine but with the gateway's. To sort the answers back to the right machines afterwards, the gateway has to tell those conversations apart — and it tells them apart the same way the kernel does, by the port number on its own side. So it hands out numbers from its range, out of the same limited field of numbers.

Hence the point: behind such a gateway the port numbers are shared across the whole network of machines. They are spent by everyone together and they run out for everyone at once. This is reasoning rather than a measurement: there is no second machine and no gateway in the run.

Two consequences worth keeping in mind. First, the absence of EADDRNOTAVAIL on the application's machine proves nothing: numbers there may well be free while connections still fail, because they ran out on the gateway and that is where the symptom comes from. Second, computing the estimate for one machine understates the risk, because behind a shared gateway the load of the machines adds up into one count rather than splitting into independent ones.

The cure, however, is the same one, and that is the good news: a connection that lives and gets reused spends no number, neither on the machine nor on the gateway.

How to answer in an interview

Short answer: TIME_WAIT is the normal end of a connection rather than a leak. The side that closed holds on to its address-and-port pair for a while, so that delayed segments of the old connection are not accepted by a new connection with the same addresses and ports. It becomes a problem for a client that opens a connection per request: what runs out is not memory and not descriptors but port numbers. Measured: with a range of 50 numbers, exactly 50 connections and EADDRNOTAVAIL on the fifty-first.

That is enough to answer correctly. Beyond it is what to add if the interviewer digs.

If the interviewer digs deeper

Three things separate a good answer. First, you give an order of magnitude rather than "it depends": the port range divided by the duration of the state is about 470 new connections per second — and you name the boundary in the same breath, that this is a rough estimate for repeated connections to one and the same destination address and port in one particular configuration, not a ceiling of the machine. Second, you do not propose turning settings but explain why they are about something else: tcp_fin_timeout applies to a different state and tcp_max_tw_buckets bounds the number of sockets rather than the duration. Third, you say that TIME_WAIT on a server means the server is closing, which is a separate conversation about why it does not keep connections.

And one phrasing worth correcting in your own head beforehand. "It goes to whoever closed first" holds for an ordinary close through FIN, but the standard's rule is stated differently: the side performing the active close is the one that goes through TIME_WAIT. On a simultaneous close both sides close actively, and a connection ended by a reset does not go through the state at all.

What not to say: "we lowered tcp_fin_timeout and it got better". If it got better, that was not the reason.

Next they ask

Common misconceptions

Practice

Two exercises. Answer first, then check against the real output: in both, the correct answer is what the measurement script prints.

server_side, client_side = who_holds(closes_first=False)
made, stopped = until_ports_run_out()
print(server_side)
print(client_side)
print(stopped)

0
20
EADDRNOTAVAIL

20
0
EADDRNOTAVAIL

0
20
EMFILE

10
10
ECONNREFUSED

Knowledge check