CoreDNS at 100K RPS: ndots, Negative Caching, and Autopath
A real-world CoreDNS latency incident at high query volume reveals how ndots and Kubernetes search domains silently multiply DNS lookups into a full-blown query storm. You'll learn: Why a single external hostname like api.stripe.com generates 4 upstream queries under Kubernetes' default ndots:5 — and how that 4x amplifier compounds at scale How to override search behavior per-pod via dnsConfig.ndots, and why a trailing dot in your FQDN collapses four queries into one Tuning the CoreDNS cache plugin: setting denial TTLs high enough that NXDOMAIN responses actually stick, and the staleness tradeoff you must articulate in interviews What the autopath plugin actually does differently — moving search-path resolution server-side — and when it helps versus when it shifts the problem Why interviewers use this question to separate engineers who've read the docs from those who've debugged a DNS storm at 3am Keywords: CoreDNS tuning, Kubernetes DNS ndots, negative caching NXDOMAIN, CoreDNS cache plugin, Kubernetes networking interview 🎧 Listen, then go deeper — DevOps & Cloud interview-prep ebooks at DevOpsInterview.Cloud ▶ Daily 30-second interview drills: DevOps Interview Cloud on YouTube Transcript Picture this. Your cluster is doing something like a hundred thousand requests per second across a few hundred services, and suddenly every outbound call to a third-party API starts timing out intermittently. Nothing in your application changed. No deploy went out. But your p99 latency for anything that touches DNS resolution just tripled, and your CoreDNS pods are pegged at max CPU. This is a real failure pattern, and it almost always traces back to two things: how ndots is configured in every pod's resolver, and how CoreDNS is caching, or failing to cache, negative responses. Interviewers ask about this because it separates people who've read the Kubernetes docs from people who've actually had to explain a production DNS storm to their team. Anyone can say 'CoreDNS handles DNS in the cluster.' Far fewer people can explain why a single hostname lookup can turn into four or five actual queries hitting your DNS servers, and what that does to your query volume when you're already running near capacity. This question tests whether you understand resolution mechanics, not just that a service called CoreDNS exists. Here's the mental model. Every pod in a Kubernetes cluster gets a resolv.conf file, and that file has two important settings: a search list and an ndots value. The search list typically looks like namespace dot svc dot cluster dot local, then svc dot cluster dot local, then cluster dot local, and depending on your cloud provider, maybe an additional domain from the node itself. The ndots value, five by default in Kubernetes, tells the resolver: if the name you're looking up has fewer than five dots in it, don't treat it as fully qualified. Instead, try appending each entry in the search list first, in order, before falling back to treating it as an absolute name. Now think about what that means for a completely ordinary external lookup, something like api dot stripe dot com. That name has two dots. Since two is less than the ndots threshold of five, the resolver assumes it might be a relative name inside your cluster. So it tries api dot stripe dot com dot your-namespace dot svc dot cluster dot local first. That fails, comes back NXDOMAIN. Then it tries api dot stripe dot com dot svc dot cluster dot local. Fails again. Then api dot stripe dot com dot cluster dot local. Fails again. Only on the fourth attempt does it try api dot stripe dot com as an absolute name, with a trailing dot, and finally succeeds. So a single application-level DNS call just became four queries hitting CoreDNS, three of which were guaranteed to fail. Now multiply that across your fleet. If you're doing a hundred thousand real DNS-triggering requests per second, and a meaningful chunk of those are external hostnames going through this same four-query pattern, your actual query volume against CoreDNS isn't a hundred thousand queries per second. It's closer to three hundred or four hundred thousand, most of it wasted work resolving things that were never going to exist in your cluster domain in the first place. That's the amplification, and it's the single biggest reason CoreDNS falls over under load that looks survivable on paper. There are two levers here, and good candidates know both. First, ndots itself. You can override the search behavior per pod using the dnsConfig field in the pod spec, setting ndots to something lower, like one, for workloads that mostly talk to external services. Or, simpler and often better, you just get your application code or your service mesh sidecar to use fully qualified domain names with a trailing dot for external calls, which bypasses the search list entirely regardless of ndots. That trailing dot is doing real work. It tells the resolver 'this is already absolute, skip the search list,' and it collapses four queries into one. The second lever is the CoreDNS cache plugin, and this is where negative caching becomes the interesting part. CoreDNS caches two kinds of answers: successful responses, and negative responses, meaning NXDOMAIN or NODATA results. By default, the cache plugin's denial caching, the negative cache, often runs with a fairly short TTL, sometimes as low as a handful of seconds depending on your Corefile. That means those three failed lookups per external hostname, the ones hitting cluster dot local variants, don't get cached long enough to actually help you. Every single request re-triggers the same failed chain. The fix is to explicitly tune the cache plugin in your Corefile. You want a reasonably generous success TTL, something like thirty seconds for internal service records that don't change often, and importantly you want to bump the denial TTL too, maybe to a similar range, so that once CoreDNS has established that api dot stripe dot com dot cluster dot local doesn't exist, it doesn't ask again for that window instead of on every single request. The tradeoff you have to talk about in an interview is staleness. If you cache negative responses for sixty seconds and then a service actually does get created with that exact name mid-window, clients will keep getting NXDOMAIN until the cache expires. For genuinely dynamic environments with frequent service creation, you tune this down. For stable production traffic patterns, you can afford to push it up and take the throughput win. This is also where autopath comes into the conversation, because interviewers like asking what it actually buys you versus just tuning ndots. Autopath is a CoreDNS plugin that moves the search path logic from the client side to the server side. Instead of the pod's resolver blindly firing off four sequential queries and waiting for each response before trying the next, autopath has CoreDNS itself walk the search list server-side and return the final correct answer in a single round trip from the client's perspective. From the pod's point of view, it looks like one query went out and one correct answer came back, even though CoreDNS internally still had to check multiple names. The honest tradeoff to mention here: autopath reduces client-observed latency and reduces the number of round trips over the network, but it doesn't eliminate the internal query cost inside CoreDNS, and it adds memory overhead per pod because it has to track pod IP to namespace mappings to know which search path applies. It also depends on the Kubernetes plugin being configured correctly, and it's had stability caveats across CoreDNS versions, so if you cite it in an interview, mention that you'd test it in a staging environment under real load before trusting it in a high-QPS production path. Combined with proper ndots tuning and cache tuning, it's a genuine option, but it's not a silver bullet you deploy blind. Let's talk about the wrong answers people give under pressure, because interviewers are listening for these. The first wrong answer is 'just scale up CoreDNS replicas.' That helps you survive the amplification, but it doesn't fix it. You're paying more compute to serve queries that shouldn't exist in the first place. The second wrong answer is 'disable DNS caching to get fresher answers.' That's backwards. You'd be removing the one thing protecting your upstream from repeated failed lookups. The third wrong answer, and this one sounds smart but isn't, is 'just increase CPU and memory limits on the CoreDNS pods.' That treats a query-volume problem as a resource problem. It buys you headroom, sure, but the moment traffic grows again, you're back in the same spiral, because the root cause, that four-to-one query amplification from ndots, is still sitting there untouched. So here's the thirty-second version. Under Kubernetes defaults, ndots five means any external hostname with fewer than five dots gets tried against your entire internal search domain before it's resolved as absolute, turning one lookup into four. Fix it by using fully qualified names with a trailing dot for external calls, or override ndots in dnsConfig for the workloads that need it. Tune the CoreDNS cache plugin's success and denial TTLs deliberately, don't just accept the defaults, because negative caching is what protects you from repeatedly re-querying names that don't exist. And consider autopath to collapse client-side search path queries into a single server-side lookup, understanding it trades some CoreDNS-side memory and complexity for lower client latency. If you want the full written version of this with the exact Corefile syntax, the dnsConfig YAML, and the follow-up questions interviewers tend to ask after this one, go grab the prep guide at devopsinterview dot cloud.
