Load balancing: who picks the server, and what they know when they pick

The short version: who picks the server, and when

Picture a supermarket with sixteen checkouts. A request is a shopper, a backend is a checkout. All of load balancing comes down to one question: who tells the shopper which queue to join, and what can that person see at the moment they give the advice.

Everything else is a matter of how early the advice is given.

DNS: advice given at the mall entrance

DNS answers before anyone else — minutes before the request even happens. It cannot see the queues, or whether a checkout is open at all. It can do one thing: hand different people different addresses, changing them from answer to answer.

That was checked against live domains. www.google.com returns eight addresses and reshuffles the order every time — the choice is left to the client. github.com returns one address at a time, but a different one each time — the name server has already chosen.

The main trap: DNS distributes address lookups, not requests. A browser asked for the address once, opened a connection, and sent a thousand requests down it — all of them to one server. So DNS is good at steering traffic between cities and poor at levelling load between machines.

A word on geoDNS — "answer with different addresses in different countries". It comes in real and imaginary varieties, and one field in the answer tells them apart. The real kind marks its answer "good only for this part of the internet"; the imaginary kind answers everyone the same. Four sites were checked: CloudFront really does return different addresses for the United States, Germany, Japan and Brazil; Bing, Microsoft and Netflix return identical ones for all four. The latter three have geography too — it is simply not done through DNS.

L4 and L7: the greeter at the door and the floor manager

L4 looks only at "from where, to where" — addresses and ports. It directs the shopper once, on the way in, and does not interfere afterwards. It is fast: one ordinary machine fills a 10-gigabit link.

L7 reads the request itself: what is being asked for, which headers, which cookie. It can route different requests from the same client differently, and retry a failed request at another checkout.

The difference people get wrong most often: L4 distributes connections, L7 distributes requests. If one client opens one connection and pushes ninety percent of all traffic through it, L4 will faithfully send that connection to a single checkout — and that checkout stalls while the other fifteen stand empty. So the two are usually stacked: L4 in front to spread connections, L7 behind to spread requests.

Algorithms: what the adviser goes by

A model was built for comparison: sixteen checkouts, each serving one person at a time while the rest wait. Below is the total time for the unluckiest shoppers — the queue plus the checkout itself (one in a hundred), in arbitrary units. Lower is better.

The important pair is the first and third rows: a fourfold difference with completely identical checkouts. The familiar "if the servers are the same, any algorithm will do" is wrong. The checkouts are the same; the shoppers are not — one has two items, the next has a full trolley. "In turn" hands out people evenly, not work.

When "the shortest queue" stops working

"Join the shortest queue" is good advice as long as the adviser can actually see the queues. There are usually many advisers. As long as each counts only its own shoppers, the advice stays good. It breaks in a different case — when they all read one shared board and the board does not update instantly.

That was checked too. While the board updates instantly, "fewest people" beats everything: 6.06. When the board lags by the time of two purchases, the same advice yields 19.95 — worse than the mindless "in turn" (17.00). The reason is plain: every adviser sees the same stale number, points at the same checkout, and creates a crowd there.

Meanwhile "look at two random ones and take the shorter" degrades over the same range only from 8.86 to 10.52. It never tries to find the best checkout, so it has nothing to lose when the data lies.

Which gives a simple rule:

• One adviser who sees everything himself → "fewest people".
• Many advisers, each counting only their own shoppers → "fewest people" still works.
• Data arrives from elsewhere and lags → "the shorter of two random ones".
• No data at all → "in turn", but then the settings need a human watching them.

Pinned sessions: what you pay for

Sometimes a shopper must always go to the same checkout — their basket is only there. That is session pinning, and it works, but it stops being balancing: what gets distributed is whole groups rather than individual shoppers.

In the same model, without pinning the unluckiest waited 6.06. With five thousand pinned groups: 26.90. With five hundred: 206.92. With fifty: eight thousand — that is, no balancing at all.

The conclusion is not that pinning is bad but that it is the price of keeping state on the server. If the basket can live somewhere shared, the price does not have to be paid.

Why this is worth knowing

A balancer looks like infrastructure you configure once. That holds right up to the first incident where "load is distributed evenly" and the latency tail has tripled — and it turns out that what was distributed evenly was requests, not work.

What follows is organised along one axis: when the decision is made, and what is known at that moment. It explains both why several layers of balancing exist at all, and why algorithms cannot be compared without asking where the balancer gets its data.

Everything stated here as a number comes from one of three places: a live DNS query, a run of the queueing model, or taken from a document linked in the sources. Numbers from the model are marked as such — they must not be read as measurements of a live cluster, and the reasons are spelled out below.

Client-side and server-side balancing

The line runs through one question: who holds the list of backends.

With server-side balancing the client knows a single address. Behind it sits a balancer that owns the list, the health checks and the algorithm. The client neither participates nor can get it wrong — but it also cannot route around the balancer once the balancer is the bottleneck.

With client-side balancing the client receives the list itself — from service discovery, from xDS, from SRV records — and chooses. There is no extra network hop, and one client's failure does not touch the others. The price is that balancing logic is now smeared across every client at once: changing the algorithm means shipping all of them. Hence sidecar proxies, which keep the list and the algorithm next to the client while letting them be updated apart from it.

There is a third case that is not usually called client-side balancing but is exactly that. When DNS returns several addresses, the client is the one choosing.

DNS: a decision made minutes early

DNS balancing is the earliest and the blindest layer. Measured against live domains, twenty consecutive queries each:

Two different strategies are visible. Google returns the whole list and shuffles the order — the decision is left to the client. GitHub returns one address at a time, but a different one; the name server has already decided and the client has nothing to choose from.

Both tricks are older than many readers. RFC 1794 described them in April 1995 and named, in the same document, the limit that has not gone anywhere:

the DNS protocol doesn't guarantee ordering

Record order is a suggestion, not an obligation. The ISP's resolver, the language's library and the operating system are all free to reorder it, and do. The deeper limit is put even more plainly:

There is no use in handing out information with TTLs of an hour, when the conditions for ordering the RRs changes minutely.

That is the whole tension of DNS balancing. For the decision to be fresh the TTL must be short. For caching to work it must be long. And even a short TTL does not save you: nobody is obliged to honour it — not the ISP's resolver, not a browser with its own cache, and certainly not a connection that is already open and will stay open for hours.

That last point deserves its own sentence, because it breaks the intuition. DNS balancing distributes resolutions, not requests. A client that opened an HTTP/2 connection and holds it made one resolution per thousand requests. What DNS balanced there was not load but the moments at which connections were established.

geoDNS, and how to check whether it is real

The idea is simple: answer with different addresses depending on where the question came from. The difficulty is that the authoritative server sees the address of the resolver, not the client — so a person in Tokyo using a public resolver in the United States gets geography computed wrong.

That is what RFC 7871 exists for: the resolver attaches part of the client's address to the query. The answer carries a SCOPE PREFIX-LENGTH — "the leftmost number of significant bits of ADDRESS that the response covers". Zero means the answer suits everyone.

So real geoDNS differs from the imaginary kind by one observable field. The check: the same question asked four times with four different client subnets substituted.

CloudFront answers with four different address sets and honestly marks the answer /24: "this answer is good for this subnet only". The other three answer identically and mark /0.

This does not mean Bing and Netflix have no geography. It means exactly one thing: they do not use EDNS Client Subnet for it. Anycast solves the same problem at the routing layer, and then one address corresponds to different machines in different parts of the world — while the DNS answer really is the same for everyone.

RFC 7871 is not enthusiastic about its own mechanism, either. It recommends turning it off by default and truncating the address to /24, because otherwise a DNS query starts saying more about where the user is than anyone needs.

L4 and L7: how much the balancer sees, and what it pays

Both sit between client and backend, and they look at different things.

L4 works on the transport: address, port, flags. It does not parse the HTTP request — often it cannot, because the request is encrypted. The decision is made once, when the connection is established, and every packet afterwards follows the path already chosen. Hence the throughput: Maglev, Google's balancer, "is able to saturate a 10Gbps link with small packets" on one ordinary machine with no special hardware.

L7 terminates the connection and reads the request. Method, path, headers, cookie — any routing criterion is available. It sees request boundaries, so it balances requests rather than connections. It can also retry a failed request on a different backend, which is impossible at L4 in principle: the connection is already established and cannot be replayed.

The difference most often missed is the unit of balancing.

At L4 the unit is a connection. A client with one long HTTP/2 connection lands on one backend and stays there forever, however many requests it sends. Ten clients, one of which sends 90% of the traffic, spread evenly across ten backends — one connection each — and one backend receives nine tenths of the work.

At L7 the unit is a request, and the same traffic spreads evenly. The price is decrypting, parsing and re-assembling every request.

Hence the usual arrangement: L4 in front, to accept connections and spread them across many L7 balancers; L7 behind, to make decisions that mean something. And that arrangement has a consequence we will return to: there end up being many L7 balancers, and each of them sees only part of the picture.

The algorithms: a model, and what it shows

What follows are numbers from a model. It is built like this: sixteen backends, each one FIFO queue with a single server. That is the classic G/G/1 setup — requests wait their turn and are handled one at a time. There is no invented "degradation formula" anywhere in it; slowdown under load appears on its own, out of waiting.

Arrivals are Poisson, service times lognormal (a heavy tail, the usual shape for web requests). The balancer sees exactly what a real one would: the number of outstanding requests and a moving average of response time. It does not know the future. The first 20,000 requests of 300,000 are discarded and results are averaged over five runs.

What this model cannot do, and must not be used to argue about: it has no network and no network latency, no dropped connections, no health checks, no backend cache (which makes a repeat request to the same server cheaper), no memory limits. It answers one question — how the distribution of requests affects queueing — and only that one.

How to read the numbers in the tables below. A cell is the request's total time in the system — how long it waited in the queue plus how long it was processed — measured in units of one average processing time. So 1.0 means "as long as an average request takes to process"; 0.83 is faster than an average processing (the request barely queued), 25.07 is twenty-five times longer. Absolute seconds are not needed here: in these units the numbers do not depend on whether the hardware is fast or slow.

The columns are percentiles, not an average. p50 is the median: half the requests came in faster than this. p95 is the 95th percentile: only one in twenty was slower. p99 is the 99th: only one in a hundred was slower. The average is useless here on purpose — a balancer is chosen by its tail, the few unlucky requests, and the tail shows up in p95 and p99, not in the mean.

Identical servers, 80% load

The first thing worth noticing: a fourfold spread on identical servers. The usual "if the servers are the same, round robin will do" is wrong, and the reason is not the servers but the requests — they differ in length. Round robin distributes them evenly by count, not by work, and a server that happens to receive three long requests in a row queues up while its neighbour idles. Least connections does not allow that, because it looks at what has not finished yet.

Random does worse than round robin for exactly the reason balls land unevenly in bins — which the section on two choices comes back to.

Weighted round robin, and the cost of a wrong weight

Now one server in sixteen is four times slower than the rest — an ordinary situation: a different hardware generation, a noisy neighbour, a degraded disk.

The numbers in the thousands are not latency; they are the absence of stability. The arithmetic is simple: sixteen servers at 80% load give each 0.8 requests per mean service time, and each request costs the slow server four times as much, so its own utilisation is 3.2. Above one. Its queue grows without bound, and the printed number says only where the simulation stopped, not how long a reader would wait.

Two more things follow. A correctly set weight restores stability but still loses to least connections by 3× at p99 — a static weight does not know what is happening right now. And a weight set slightly wrong is the worst option of all: p95 looks healthy (16.13), and only p99 reveals that the server is balanced on the edge. Monitoring by mean and p95 will not catch that configuration.

Least connections and least response time

nginx describes least connections in so many words: the request goes "to the server with the least number of active connections, taking into account weights of servers". Least response time adds time to that; in nginx it is least_time, with a choice of measuring to the header (header) or to the last byte (last_byte).

In the model above the two are close: 6.06 against 5.50 at p99 on identical servers, and 7.70 against 6.48 on the cluster with a slow one. Least response time is slightly better, which is expected — a connection count cannot tell a fast backend from a slow one, and time can.

The gap widens where connection count stops being a measure of work at all. A connection streaming a gigabyte file and a connection carrying an empty health check look identical to least connections.

Least bandwidth and least packets

These are about the link, not the queue. NetScaler defines least bandwidth as choosing the service "that is currently serving the least amount of traffic, measured in megabits per seconds", and least packets as "the least packets in the last 14 seconds".

Measure whatever runs out first. For an API whose response is a kilobyte of JSON, CPU runs out, and queues are the right measure. For video delivery bandwidth runs out, and there least connections misleads: ten connections at 100 kbit/s and one at 50 Mbit/s are not "ten against one", they are "one against fifty".

There is deliberately no model for these two here: building one would mean modelling the network rather than the queues, and the result would say more about assumptions regarding the network than about balancing. So this section carries the vendor's definition and the rule for choosing, and no numbers.

Sticky sessions: what pinning costs

Pinning a session to a backend is needed when state lives on the server. The price is that the balancer stops balancing: it now distributes sessions, not requests.

Even at five thousand sessions p99 is four and a half times worse. Beyond that the curve collapses: the fewer the sessions, the coarser the grain the load is laid out in, and the stronger the random skew. At fifty sessions across sixteen servers balancing has simply ceased to exist — the law of small numbers has taken over.

Hence a practical rule worth more than any tuning: session pinning is compensation for server-side state, not a way to balance load. If the state can move to shared storage or into a signed cookie, no pinning is needed at all. If it cannot, it is worth knowing that the tail latency is what paid for it.

Technically the pinning is done in several ways, and the differences matter. nginx's ip_hash takes "the first three octets of the client IPv4 address" — so every client behind one NAT lands on one backend. Cookie-based pinning is more precise but only works for browsers. Consistent hashing (hash ... consistent in nginx, ring hash and Maglev in Envoy) differs from a plain hash in that adding or removing a server moves a small fraction of the keys rather than nearly all of them; nginx's own documentation says it outright — "only a few keys will be remapped".

Power of two choices: why two, and not all of them

Start with the pure problem: n balls thrown into n bins. If each ball flies into a random bin, the maximum load "is approximately log n / log log n". If each ball looks at two random bins and drops into the emptier one, it becomes "log log n / log d + O(1)". Simulated with balls and bins — fifteen runs per n, mean and standard deviation:

A cell is the maximum load: how many balls ended up in the fullest bin (mean over fifteen runs; the number after ± is the standard deviation). n is both the number of balls and the number of bins. The first two columns are the two ways to throw a ball — into a random bin, and into the emptier of two random ones. The last two are the asymptotic formulas from the survey, computed with the natural logarithm, for comparison.

A thousandfold increase in bins nearly doubles the maximum under random choice — and barely moves it under a choice of two: 2.60 → 3.53. The asymptotic formula underestimates the random case at finite n (it is asymptotic; that is expected), while the two-choice case stays close to the model and converges with it at a hundred thousand bins: 3.53 against 3.53.

Why two rather than three or ten is answered in the same survey: "each additional choice beyond two decreases the maximum load by just a constant factor". The first step, from one to two, changes the asymptotics; every step after it changes only a constant. HAProxy encoded exactly that: the number of draws is configurable and defaults to two. Envoy words it the same way: "selects N random available hosts as specified in the configuration (2 by default) and picks the host which has the fewest active requests".

But in queues rather than bins the picture is different — and this is the surprise of the whole article. Look back at the first table: p99 for power of two is 8.86, for least connections 6.06. The two-choice pick loses. And this is not an artefact of the model: HAProxy's author, measuring on a live rig of six machines, got the same sign — least connections delivered "about 4% higher" requests per second, and its connection peaks were "about 30% lower".

So what is power of two for?

The headline result: signal freshness beats algorithm

Least connections needs to know how many requests each backend currently has in flight. While there is one balancer it knows this exactly — it sent them. Once there are many balancers, nobody knows exactly.

The usual explanation is the herd effect: every balancer sees the same "least busy" server and rushes at it together. Owen Garrett describes it through immigration queues: "all the guides notice that one queue is momentarily shorter and faster, and all send travelers to that queue". Envoy names the same cause as its reason for choosing P2C: "resistance to herding behavior".

The explanation is persuasive, so it was tested with two experiments. It turned out to be half right.

Experiment one: every balancer keeps its own counters. Sixteen backends, several independent balancers; each sees only the requests it sent and knows nothing about its neighbours.

Cells are p99, in the same units of one average processing time as above; the row is how many independent balancers run at once.

There is no herd. Least connections loses its advantage smoothly and by sixty-four balancers matches round robin — 18.38 against 19.72 — but it never becomes worse than round robin. The reason is clear once stated: a private counter is an unbiased sample of the whole picture. The balancer judges on incomplete data, not on wrong data, and the errors of different balancers do not conspire.

Experiment two: a shared snapshot with a lag. Now the balancers read one shared picture, refreshed once per refresh (in units of mean service time) — the way setups work when backend metrics are collected and distributed.

Cells are p99 again; the row is how stale the shared snapshot is: 0 is instant, 2.0 lags by two mean service times.

Now everything is visible. At a lag of two mean service times, least connections gives p99 = 19.95 — worse than round robin and nearly level with random choice. Power of two moves from 8.86 to 10.52 over the same range, about a fifth.

So it is not the number of balancers as such. What destroys least connections is a shared stale signal: everyone reads the same wrong number and makes the same mistake at the same moment. Private incomplete counters do not do that; a shared snapshot does, and the older it is the worse it gets.

Hence a rule stated in terms of data rather than algorithm names:

• One balancer, counting for itself — least connections or least response time, and nothing better is needed.
• Many balancers, each counting its own requests — least connections is still sensible; it degrades toward random, never below it.
• State arrives from elsewhere, with a lag — pick the better of two random servers, and the larger the lag the more obvious the win.
• No state at all — weighted round robin, and then the weights need a human watching them; a wrong weight costs more than it looks.

When there is nothing left to balance: shedding load

The model above showed the point past which the algorithm decides nothing: utilisation above one, and the queue grows without bound. That is not the edge of the model but an operating condition, and it has an answer of its own, unrelated to balancing.

Avoiding overload is a goal of load balancing policies. But no matter how efficient your load balancing policy, eventually some part of your system will become overloaded. Gracefully handling overload conditions is fundamental to running a reliable serving system.

The answer is called load shedding, and its definition is short:

Load shedding drops some proportion of load by dropping traffic as the server approaches overload conditions.

Why refuse rather than queue is explained by the other half of our own model. A queue is not free waiting but spent memory and added latency:

Queued requests consume memory and increase latency. For example, if the queue size is 10x the number of threads, the time to handle the request on a thread is 100 milliseconds. If the queue is full, then a request will take 1.1 seconds to handle, most of which time is spent on the queue.

And, above all, work in such a queue is no longer wanted by anyone: If a user's web search is slow because an RPC has been queued for 10 seconds, there's a good chance the user has given up and refreshed their browser, issuing another request: there's no point in responding to the first one, since it will be ignored. Past the threshold, the server spends resources on answers nobody will read. Shedding returns them to whoever is still waiting:

The goal of load shedding is to keep latency low for the requests that the server decides to accept so that the service replies before the client times out.

The mechanics. They use exactly the quantity least connections already operates on in the table above — the number of requests in flight:

One straightforward way to shed load is to do per-task throttling based on CPU, memory, or queue length […] For example, one effective approach is to return an HTTP 503 (service unavailable) to any incoming request when there are more than a given number of client requests in flight.

In Envoy these are two different mechanisms, and they are worth telling apart. Circuit breaking — max_requests and max_pending_requests per cluster — protects the party being called. The overload manager protects the party in the middle: This is distinct from circuit breaking which is primarily aimed at protecting upstream services. The latter's thresholds are set by resource pressure: drain connections at 92 % heap use, stop accepting requests at 95 %.

The threshold need not be chosen by hand. Adaptive concurrency derives it from latency: it measures the ideal round-trip time minRTT, compares it with the current one and moves the limit by a gradient — This gradient value has a useful property, such that it decreases as the sampled latencies increase. The price is named in the same place: during the measurement window there may be a noticeable rise in 503s, because the limit is pinned to its minimum for that time.

Two caveats, without which the advice does harm.

The first is about the metric. Counting capacity in requests per second is unreliable: modeling capacity as "queries per second" … often makes for a poor metric. For the same reason our model gives: requests are not equal in cost, and it was the spread of cost, not their number, that grew the tail.

The second is that the shedding path must itself work. A path nobody uses is usually broken:

Remember that the code path you never use is the code path that (often) doesn't work. In steady-state operation, graceful degradation mode won't be used, implying that you'll have much less operational experience with this mode and any of its quirks, which increases the level of risk.

Retries: why "try again" multiplies

In the L4-versus-L7 section above, retrying on another backend is listed as an advantage of L7 — and it is one. What was not said is the other half: under overload, a retry stops helping and starts harming.

When failures are caused by overload, retries that increase load can make matters significantly worse. They can even delay recovery by keeping the load high long after the original issue is resolved.

The dangerous part is that retries multiply across layers rather than adding up:

Avoid amplifying retries by issuing retries at multiple levels: a single request at the highest layer may produce a number of attempts as large as the product of the number of attempts at each layer to the lowest layer. If the database can't service requests because it's overloaded, and the backend, frontend, and JavaScript layers all issue 3 retries (4 attempts), then a single user action may create 64 attempts (4^3) on the database.

AWS works the same multiplier over five layers and gets 243: If each layer retries independently, the load on the database will increase 243x, making it unlikely to ever recover. Both numbers are arithmetic rather than measurement, and that matters more than a measurement here: the exponent is the number of layers, and nobody counts the layers in a real system.

Put this together with the model above. Utilisation reached one, latency began to climb, clients started retrying — and the load on the bottom layer grew not by percentages but several times over: the multiplier here is raised to the power of the number of layers, not multiplied by it. Three layers of three retries is 64 attempts instead of one; five layers is 243. The point of no return, which looked distant on the graph, sits behind that multiplier.

Three measures. Two are named outright by the sources; the third follows from them.

Spread retries out in time. Always use randomized exponential backoff when scheduling retries. The randomness is not decoration: without it retries synchronise and arrive in a burst — Jitter adds some amount of randomness to the backoff to spread the retries around in time.

Bound retries with a budget, not a per-request counter. Consider having a server-wide retry budget. For example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don't retry; just fail the request. The difference is fundamental: a per-request counter bounds one client, a budget bounds all of them at once, and it is the total volume that multiplies.

Retry on one layer only. That follows from the first quote, and in Envoy's configuration it is written like this:

In general we recommend using retry budgets; however, if static circuit breaking is preferred it should aggressively circuit break retries. This is so that retries for sporadic failures are allowed, but the overall retry volume cannot explode and cause large scale cascading failure.

Proximity against evenness: zone-aware routing

The whole model above silently treated the backends as equally reachable. Across several zones that is untrue: a request into its own zone has lower latency, and cross-zone traffic is billed separately besides. This conflicts with the very thing a balancer is for, and Envoy states the conflict without smoothing it over:

The purpose of zone aware routing is to send as much traffic to the local zone in the upstream cluster as possible while roughly maintaining the same number of requests per second across all upstream hosts (depending on load balancing policy).

"As much as possible" and "roughly the same" are two goals, and the second bounds the first. What happens when they diverge is described concretely:

The originating cluster local zone percentage is greater than the one in the upstream cluster. In this case we cannot route all requests from the local zone of the originating cluster to the local zone of the upstream cluster because that will lead to request imbalance across all upstream hosts. Instead, Envoy calculates the percentage of requests that can be routed directly to the local zone of the upstream cluster. The rest of the requests are routed cross zone.

So locality is not a switch but a fraction, and the fraction is computed from the ratio of the zones' capacities.

The second approach is built the other way round and is incompatible with the first. Locality weights are supplied not by the balancer's heuristics but by the management server:

This approach is mutually exclusive with zone aware routing, since in the case of locality aware LB, we rely on the management server to provide the locality weighting, rather than the Envoy-side heuristics used in zone aware routing.

What matters here is how the weights behave under failure. A weight is not adjusted immediately but with slack — an over-provision factor of 1.4 — and the table from the documentation shows where the spill-over begins (locality X with weight 1 against Y with weight 2):

Between 100% and 70% nothing changes: the 1.4 factor is precisely "while fewer than 30% are unavailable, treat the locality as whole". Past that the share falls proportionally. These are Envoy's documented numbers, not our measurement.

And, for this article's main thread, the point about when the decision is made. The algorithm from the table above picks a host not first but third:

1. Pick priority level. 2. Pick locality (as described in this section) within priority level from (1). 3. Pick endpoint using cluster specified load balancer within locality from (2).

So the "round-robin or least connections" argument the model is devoted to plays out inside an already-chosen locality. The two preceding steps cut away most of the hosts before the algorithm gets a say at all — and they shape the final distribution more than the choice between it and its neighbour does.

A little history

What is a standard here, and what is one product's decision

The distinction is the same kind as "language guarantee versus implementation detail", and confusing the two is just as dangerous.

Standards you can lean on: the DNS answer format and TTL semantics (RFC 1035, RFC 1794); the EDNS Client Subnet format and the meaning of SCOPE PREFIX-LENGTH (RFC 7871); the theoretical bounds for random choice and for the best of two — that is mathematics, and it does not depend on a product.

One product's decision, which changes between releases: which algorithms exist and what they are called; what exactly nginx counts as an "active connection"; how many draws HAProxy makes by default (two — but it is a setting); Envoy's formula for unequal weights; the fact that ip_hash takes the first three octets; Maglev's table size (65537). None of these numbers is a property of load balancing itself — every one of them was chosen by an author and written down in documentation worth citing when the argument starts.

Reproducing the numbers

Every number here comes from one of three places.

The live DNS measurements are ordinary resolver queries, with EDNS Client Subnet substituted for four subnets; the result depends on where you are and which resolver you use, so your addresses will differ — but the scope field will not, and that is the field being tested. The scripts: bench/load-balancing/dns2.py and bench/load-balancing/ecs.py.

The queueing model is the discrete-event simulation described above: sixteen FIFO queues, Poisson arrivals, lognormal service times, 300,000 requests, the first 20,000 discarded, five runs with different seeds. Algorithms are computed by bench/load-balancing/sim2.py, weights and affinity by bench/load-balancing/sim3.py, many balancers at once by bench/load-balancing/herd.py and bench/load-balancing/herd2.py.

The balls-and-bins problem is bench/load-balancing/bins.py.

Model numbers reproduce exactly; the DNS numbers reproduce only in kind.