What Is DNS Resolution? How It Actually Works for DevOps Engineers

DNS is one of those fundamentals that every DevOps engineer uses daily but surprisingly few can explain clearly when it breaks in production.

Understanding DNS resolution properly — not just "it converts domain names to IP addresses" — makes debugging production issues significantly faster. Let me walk through how it actually works.

The Journey of a DNS Query

When your browser (or kubectl, or curl) needs to resolve api.example.com, here's what happens:

Step 1: Check the Local Cache

The operating system first checks its DNS cache. If you visited api.example.com recently and the cached result hasn't expired, it returns the IP immediately without any network call.

# View DNS cache on Linux (systemd-resolved)
systemd-resolve --statistics
 
# Flush DNS cache
sudo systemd-resolve --flush-caches
 
# On Mac
sudo dscacheutil -flushcache

Step 2: Check /etc/hosts

If not cached, the OS checks /etc/hosts. Entries here override DNS entirely.

cat /etc/hosts
# 127.0.0.1 localhost
# ::1       localhost
# 192.168.1.100 my-dev-server  # Custom entry

This is why kubectl port-forward plus /etc/hosts entries is a common development workaround — it lets you route production-like domain names to local services.

Step 3: Query the Recursive Resolver

If not in cache or hosts, the OS sends a query to the recursive resolver — typically configured by your ISP, company network, or manually set to 8.8.8.8 (Google) or 1.1.1.1 (Cloudflare).

The recursive resolver does the heavy lifting: it queries the DNS hierarchy and returns the answer to your machine.

Step 4: The Recursive Resolver Walks the Tree

If the recursive resolver doesn't have the answer cached, it starts from the top of the DNS hierarchy:

1. Root nameservers (.) — There are 13 root nameserver clusters worldwide. The resolver asks: "Who handles .com?"
2. TLD nameservers (.com) — The root says "ask ns1.verisign.net." The resolver asks: "Who handles example.com?"
3. Authoritative nameserver (example.com) — The TLD nameserver says "ask ns1.cloudflare.com." The resolver asks: "What's the IP for api.example.com?"
4. Final answer — The authoritative nameserver returns the A record: api.example.com → 203.0.113.45

This whole process takes 50-200ms on a cold cache. With caching it's microseconds.

Understanding TTL

Every DNS record has a TTL (Time To Live) — a number in seconds that tells resolvers how long to cache the answer.

# Check TTL for a domain
dig api.example.com
 
# Output shows:
# api.example.com. 300 IN A 203.0.113.45
# The "300" is the TTL in seconds (5 minutes)

Why TTL matters in production:

When you change a DNS record (after a migration, failover, or IP change), the change propagates gradually. Clients and resolvers with the old IP cached will keep using it until their cached entry expires.

If you're planning a migration, set TTL low (60-300 seconds) several hours in advance. After the migration, raise it back to a higher value (3600-86400).

Low TTL = faster propagation = more DNS queries = slightly more load on your nameservers. High TTL = slower propagation = fewer DNS queries = less nameserver load.

DNS Record Types You Need to Know

Common mistake: pointing a CNAME to an IP address. CNAME must point to another domain name, not an IP. Use an A record for IPs.

DNS in Kubernetes: CoreDNS

Inside a Kubernetes cluster, DNS works differently. Every cluster runs CoreDNS — a DNS server that handles service discovery within the cluster.

When a pod does nslookup postgres-service, CoreDNS resolves it to the ClusterIP of the postgres-service Service object.

The DNS name format in Kubernetes:

<service-name>.<namespace>.svc.cluster.local

So postgres-service in the production namespace is fully addressable as:

postgres-service.production.svc.cluster.local

Pods in the same namespace can just use postgres-service. Cross-namespace calls need the full name.

Check CoreDNS is running:

kubectl get pods -n kube-system -l k8s-app=kube-dns

Debug DNS from inside a pod:

# Run a debug container
kubectl run dns-debug --image=busybox:1.35 --restart=Never -it -- /bin/sh
 
# Inside the container:
nslookup kubernetes.default
nslookup postgres-service.production.svc.cluster.local
cat /etc/resolv.conf

Debugging DNS Issues in Production

Tools for DNS Debugging

# Basic lookup
nslookup api.example.com
 
# Detailed query with timing
dig api.example.com
 
# Trace full resolution path
dig +trace api.example.com
 
# Query specific nameserver
dig @8.8.8.8 api.example.com
 
# Check what nameservers are authoritative
dig NS example.com
 
# Check DNS propagation for a recent change
dig api.example.com @ns1.cloudflare.com  # Check primary NS
dig api.example.com @8.8.8.8             # Check Google's resolver
dig api.example.com @1.1.1.1             # Check Cloudflare's resolver

Common Production DNS Scenarios

"It works on my machine but not in production"

Usually a TTL/propagation issue. Your machine cached the new record. Production servers still have the old one. Wait for TTL to expire, or flush DNS cache on affected servers.

Kubernetes pods can't resolve external hostnames

Check CoreDNS:

kubectl logs -n kube-system -l k8s-app=kube-dns
 
# Also check the pod's resolv.conf
kubectl exec -it <pod-name> -- cat /etc/resolv.conf

If CoreDNS is crashing, check for high cardinality metrics or a buggy CoreDNS plugin. Restart CoreDNS if needed:

kubectl rollout restart deployment coredns -n kube-system

DNS resolution is slow

Check your resolver response time:

dig @8.8.8.8 example.com | grep "Query time"

If queries take >100ms consistently, consider running a local caching resolver (dnsmasq, systemd-resolved with caching) or switching to a faster public resolver.

The One Concept That Changes How You Debug

Most DNS confusion comes from not understanding caching at multiple layers:

1. Browser cache
2. OS cache
3. Your company's recursive resolver cache
4. The public recursive resolver cache (8.8.8.8, 1.1.1.1)

When you change a DNS record and it "doesn't propagate," it usually has propagated at the authoritative nameserver — it's still cached somewhere in the chain. Understanding which cache needs to expire (and what its TTL is) tells you exactly when the change will take full effect everywhere.

More networking fundamentals? Read our TCP/IP stack explained simply and Kubernetes CoreDNS resolution failures fix.