Name resolution: the application calls getaddrinfo, not DNS

What this question is really about

The ladder looks like this:

1. "How does an application find an address for a name?" — the warm-up, where "it goes to DNS" is the answer that suggests itself.
2. "What happens if a name is in both /etc/hosts and DNS?" — a question about the order of sources.
3. "Where is the result cached?" — a trap: on the direct path, nowhere.
4. "Our timeout is 300 ms and a request hung for 12 seconds. How?" — the central question of the topic.
5. "Why do names resolve more slowly inside a cluster than outside it?" — a question about search domains and ndots.
6. "What do you do when a name server stops answering?" — a question about there being no such thing as a spare server.

The numbers come from running bench/nameres/resolve.py and bench/nameres/practice.py. Real query times in this environment wander between 2.2 and 29.3 ms, so no claim here rests on them: what carries meaning is the count of queries and the delays the library counts off on a clock.

Base: name, address, connection

The model everything else follows from is three words long: name → address → connection. Code and configuration hold a name; a connection is ultimately established with a numeric address; so there has to be a separate translation step in between. That step happens before any connection exists at all — and that is not a detail but the cause of half the surprises in this topic.

"Who translates a name into an address" is usually answered with "DNS", and the answer is incomplete in one important place: the application does not talk to a name server itself. It asks the operating system — one request, "give me the address for this name" — and the system decides where to get the answer from. Behind that decision stands an order of sources: a local file on this same machine first, and only if nothing was found there, a name server over the network.

Three consequences follow immediately, and they are what the lesson is for. The answer can arrive with no network involved, in which case there is no DNS in the story at all. The duration of this step is set not by somebody else's server speed but by this machine's settings — who to ask, how long to wait, how many times to retry. And finally, the timeout you set on your request does not apply to this step: it is set on a connection, and there is nothing to connect to yet.

That system function is called getaddrinfo on Linux, and from here the lesson answers the question of how the OS does it: where the order of sources comes from, what the price of an answer is made of, and why one such call can cost seconds.

That is already enough to answer the basic interview question. Everything below is the numbers: what an answer from a file and an answer from the network cost, what a name that does not exist costs, and why the same name resolves more expensively inside a cluster than on a laptop.

Mechanism 1: the answer can arrive without a network

"How much does DNS cost" is the wrong question to begin with. The application calls getaddrinfo, and where that goes for an answer is decided by a separate file:

The line hosts: files dns reads literally: /etc/hosts first, and only if nothing was found there, the name server. What that looks like:

1. THE ANSWER CAN COME WITHOUT ANY NETWORK AT ALL
-------------------------------------------------
  nsswitch.conf says hosts:                    files dns
  getaddrinfo('localhost'), ms                 0.01  -> 127.0.0.1
  getaddrinfo('example.com'), ms               29.32  -> 104.20.23.154

The same call, the same library — and a completely different price, because the first name never left the machine.

Two practical consequences follow, and they are what this mechanism gets asked about. First: an entry in /etc/hosts beats DNS silently, and nobody will see it in a packet trace — there was no network request. Second: when an application in a container talks to the wrong place, the image's /etc/hosts is the first place to look, not the last.

Mechanism 2: on the path examined here there is no cache

The misconception this section exists for: "we resolved it once, after that it comes from the cache". Six consecutive calls settle it.

2. THERE IS NO CACHE INSIDE THE PROCESS
---------------------------------------
  call 1 of getaddrinfo('example.com'), ms     2.17  -> 104.20.23.154
  call 2 of getaddrinfo('example.com'), ms     24.06  -> 104.20.23.154
  call 3 of getaddrinfo('example.com'), ms     2.15  -> 172.66.147.243
  call 4 of getaddrinfo('example.com'), ms     2.41  -> 104.20.23.154
  call 5 of getaddrinfo('example.com'), ms     14.82  -> 104.20.23.154
  call 6 of getaddrinfo('example.com'), ms     2.17  -> 104.20.23.154
  distinct addresses returned                  2
  last call against the first, ms              2.17 against 2.17

Two lines matter. The last call costs exactly what the first did: 2.17 ms against 2.17, although four identical calls happened in between. The spread inside the series — 2.15 to 24.06 ms — has nothing to do with the call number. The address is not stable: it changed on the third call, and six calls returned two different ones.

Both observations say the same thing: on this path the result is not kept anywhere.

The boundary of that conclusion has to be named right away, because it is narrower than it sounds. What was measured is the absence of a cache on one specific path: direct getaddrinfo calls in this environment, with its hosts: line and its resolver. The measurement does not say there is no cache anywhere — it never touched any of the places where a cache may perfectly well sit:

• in the application itself or its runtime — client libraries and virtual machines quite often keep a name cache of their own;
• in another NSS source — the hosts: line can hold more than files and dns;
• in a system caching resolver on the same machine — nscd, systemd-resolved;
• in a cluster's node-level cache, which a container's resolver may point at;
• in a sidecar process next to the application, through which its traffic runs.

So the rule is written from the boundary rather than from the measurement: "does a name get cached here" is a question about the environment rather than about the library, and it is answered by checking your own path, not by assuming a default. There is no default worth relying on.

The practical point: a client library that resolves a name per request pays for it on every request in an environment with no cache. Hence the habit of keeping a connection rather than opening a new one: a reused connection not only saves the handshake, it asks for no name at all.

And hence caution about the opposite move. An in-process name cache looks like an obvious optimisation right up to the day the address changes: a program with its own cache will keep going to a machine that is gone until it is restarted, while a program without one notices the change on the next call.

Mechanism 3: the price of a missing name is arithmetic

Three settings set the price: timeout, attempts and the number of nameserver lines. The entry on timeout warns about how it ends by itself:

"No guarantee that a single resolver API call maps to a single timeout" is the warning. How many timeouts one call actually costs is what the measurement shows:

3. A NAME THAT DOES NOT RESOLVE COSTS timeout x attempts x servers
------------------------------------------------------------------
   nameservers  timeout  attempts  expected, s  measured, ms  queries
             1        1         1            1        1001.4        1
             1        1         2            2        2002.7        2
             1        1         3            3        3003.7        3
             1        2         2            4        4004.2        2
             2        1         2            4        4003.8        4

The "expected" column was computed by multiplication before the run, the "measured" column is the result. They agree to within a few milliseconds: in this resolver configuration the price of a name that does not resolve came out as timeout × attempts × number of servers, not "some amount".

The boundary of that equality has to be named right away, or it turns into a universal formula, which it is not. The multiplication here is not a law but the result of one particular setup: a single glibc resolver, one source in the hosts: line, and a name server that accepts queries and answers none of them. resolv.conf(5) warns about this itself, in the line quoted above: there is "no guarantee that a single resolver API call maps to a single timeout". Change any of those conditions — a different resolver, a server that refuses instead of staying silent, a caching intermediary on the way, a search list — and the factors stop multiplying out to this product.

So the rule is written from the boundary rather than from the formula: the duration of this step is set by the resolver's settings rather than by the network, and you learn it by computing it from your own resolv.conf and confirming it by measurement, not by taking a product of three numbers on faith. What the product is good for is the order of magnitude: seconds, not milliseconds.

The last row answers the question of what a second server buys. The measurement gives the factor: two servers, four queries instead of two. That the second one is tried after the first rather than instead of it comes from resolv.conf(5), in the line quoted above: the resolver waits the full timeout "before retrying the query via a different name server".

Now take the settings a container usually has: timeout:2 attempts:3 and two servers. Measured with those, one name that does not exist costs 12.0 seconds.

Where those twelve seconds land is a separate question, and the answer is not in the measurement but in the signature: getaddrinfo has no timeout parameter at all. A timeout set on a socket does not bound this part — there is nothing to connect to yet. Only a deadline covering the whole call can.

That is the answer to "our timeout is 300 ms and it hung for 12 seconds": the timeout was on the connection.

Mechanism 4: the default asks twice

A short section that explains a common shape of "sometimes slow, sometimes not": the call waits for two answers and one arrives. getaddrinfo(3) says what happens when no hints are given:

And AF_UNSPEC means "any family": IPv4 and IPv6 both. That is two questions instead of one:

4. AF_UNSPEC ASKS TWICE: ONE QUESTION PER ADDRESS FAMILY
--------------------------------------------------------
  AF_UNSPEC (the default of most clients)      2 queries, 1001.3 ms
  AF_INET (IPv4 only)                          1 query, 1001.3 ms

The two rows took almost the same time, and that is not an accident: both questions go out together and the call waits for them in parallel. The second question does not double the time — it doubles the number of parties the answer depends on.

From there — no longer from the measurement but from the fact that the call waits for both — comes an unpleasant class of failure: a server that answers A instantly and stays silent on AAAA makes the whole call slow, although the answer that was needed had already arrived. From the application's side this looks like "DNS is sometimes slow", and time alone will not tell the cause.

The diagnostic follows: if one and the same name resolves quickly with an explicit AF_INET and slowly by default, the problem is not the network but the second question.

A caveat from the same quotation: AI_ADDRCONFIG can remove the second question — on a machine with no global IPv6 address of its own. The measurement environment has one, which is why the run shows two queries rather than one.

Deeper: search domains multiply the query

The last section is about why names resolve differently inside a cluster than on a laptop. resolv.conf(5) describes the mechanism and warns about its price itself:

ndots owns the threshold: a name with fewer dots than the threshold counts as incomplete, and the search domains get appended to it first. What actually gets asked:

5. SEARCH DOMAINS TURN ONE LOOKUP INTO SEVERAL
----------------------------------------------
  ndots:5, name has one dot                    2002.5 ms, 2 lookups
    asked for                                  zz-probe-9931.internal.svc.cluster.local
    asked for                                  zz-probe-9931.internal
  ndots:1, same name                           2002.9 ms, 2 lookups
    asked for                                  zz-probe-9931.internal
    asked for                                  zz-probe-9931.internal.svc.cluster.local
  ndots:5, same name with a trailing dot       1001.4 ms, 1 lookup
    asked for                                  zz-probe-9931.internal

Three rows, three different answers to the same question of "what address does a name ending in .internal have".

Under ndots:5 the name has one dot, the threshold is not met, so the name with the search domain appended is asked for first — the one that certainly does not exist. The real name is tried second.

Under ndots:1 the threshold is met and the order reverses: the name as written first, the search domain second.

With a trailing dot the name is declared complete and the search list is never touched: one query instead of two, exactly half the price.

Combining this with mechanism 3: every extra name goes through the whole timeout × attempts × servers chain again. That combination was not measured — it is arithmetic: with the environment's defaults and three names tried, it is three times twelve seconds.

How to answer in an interview

Short answer: the application does not call DNS but a function of the operating system, and behind that function stands an order of sources — the file first, the name server after it. So the answer can arrive with no network at all, and the duration of this step is set by the machine's settings and is not bounded by the request's timeout: the timeout is set on a connection, and there is nothing to connect to yet.

That is enough to answer correctly. Beyond it is what you add when the interviewer digs.

If the interviewer digs deeper

The function is getaddrinfo, the order of sources comes from nsswitch.conf and the retry schedule from resolv.conf. Measured: a name from the file took 0.01 ms and sent no packet; in this resolver configuration a name that does not exist cost timeout × attempts × number of servers, second for second.

Three things separate a good answer. First, you say that getaddrinfo has no timeout parameter: a timeout set on a socket does not bound this part, because there is nothing to connect to yet. Second, you call the second name server a doubling of the worst case rather than a spare, and explain why. Third, you mention that the direct path through getaddrinfo turned out to have no cache, and draw the right conclusion from it: not "let's add a cache" but "let's not resolve a name per request".

Two caveats separate a precise answer from a memorised one. First, "there is no cache" is a claim about the path that was examined, not about the world: a cache can sit in the application itself or its runtime, in another NSS source, in a system caching resolver, in a cluster's node cache or in a sidecar process next to the application — and whether you have one is a question about your environment. Second, timeout × attempts × servers is not a universal formula but how the price came out under this resolver configuration against a server that stays silent; what you change is your own resolv.conf, and you confirm it by measuring rather than by trusting the product.

What not to say: "DNS answers in about so many milliseconds". That number means nothing: it depends on the server, on caching, and on whether you asked for one address family or two.

Next they ask

Common misconceptions

Practice

Two exercises. Answer first, then check against the real output: in both, the correct answer is what the measurement script prints.

relative = asked_for(server, f"{PROBE}.internal", ndots=5)
absolute = asked_for(server, f"{PROBE}.internal.", ndots=5)
families = family_queries(server)
print(len(relative))
print(len(absolute))
print(families)

2
1
2

1
1
1

2
2
1

3
1
2

Knowledge check