Diagnosing CoreDNS Failures and Containerd Overlay Mount Issues in Kubernetes on Ubuntu 22.04

CoreDNS CrashLoop on Ubuntu 22.04 – DNS Resolution Failure

Recently, I encountered a perplexing DNS outage in our Kubernetes cluster running on Ubuntu 22.04 nodes. Kubernetes CoreDNS pods – which provide internal DNS for the cluster – were repeatedly CrashLooping or getting stuck in a terminating state. This meant that DNS lookups within the cluster were failing (“no servers could be reached” errors on nslookup/dig), and many applications started logging DNS resolution failures. Strangely, the usual suspects for pod crashes (out-of-memory kills, CPU pressure, etc.) were absent, and other system pods (API server, controller manager, etc.) were healthy. The cluster appeared fine, yet name resolution was dead in the water.

Initial Investigation

My first thought was to check for known CoreDNS issues. One common pitfall on Ubuntu is a DNS configuration loop (e.g. if /etc/resolv.conf in the CoreDNS pod points to the CoreDNS service itself, triggering the loop plugin error). However, in this case CoreDNS wasn’t reporting any loop errors – it was simply failing to start. I also verified the cluster’s network plugin was fine and that the CoreDNS configuration hadn’t been tampered with. Nothing obvious jumped out, which made me suspect something environment-specific with Ubuntu 22.04’s defaults might be at play.

Clue in the Logs

Each CoreDNS pod crash produced a log message: “listen tcp :53: bind: permission denied”. This was a huge clue. It indicated CoreDNS was unable to bind to port 53 (the DNS port) on the node. Normally, if a port is already in use, you’d expect an “address already in use” error; “permission denied” suggested a lack of privileges or a restriction. CoreDNS runs as a non-root user in our setup, so binding to a low-numbered port like 53 could fail unless special permissions (capabilities) are granted. But what changed on these Ubuntu nodes to cause this now?

Root Cause – systemd-resolved vs CoreDNS (Port 53 Collision)

We soon uncovered the culprit: Ubuntu’s systemd-resolved service. By default, Ubuntu (18.04+ including 22.04) enables systemd-resolved as a local DNS stub resolver. This service runs on the host and listens on 127.0.0.53:53 (port 53 on the loopback interface). In Ubuntu 22.04, systemd-resolved is typically running out-of-the-box and even replaces the system’s /etc/resolv.conf with a stub file that points to 127.0.0.53. In our cluster, the CoreDNS pods attempted to bind to port 53 (they operate on that port to serve DNS to the cluster). However, because systemd-resolved on the host was already bound to port 53 on localhost, the CoreDNS containers (which do not run as root) could not bind to that port. Essentially, the host’s DNS stub was occupying the port and our CoreDNS lacked permission to override it. The result: CoreDNS failed to start with “permission denied” and kept crash-looping, leaving the entire cluster without DNS service.

Resolution

Once we understood the root cause, the fix was straightforward:

• Disable the Ubuntu stub resolver: In /etc/systemd/resolved.conf set DNSStubListener=no and provide an upstream DNS server. Then disable the systemd-resolved service entirely with sudo systemctl disable systemd-resolved --now.

• Correct kubelet’s resolv.conf: Point kubelet to the real DNS resolvers via --resolv-conf=/run/systemd/resolve/resolv.conf, or directly update /etc/resolv.conf to bypass the stub.

With these changes, CoreDNS successfully started and resumed normal DNS resolution.

Kubernetes Node Failure – Missing Containerd Overlay Mounts

Another incident hit when a node rebooted and lost its mount for ephemeral storage under /mnt/data. This mount hosted containerd’s working directory /mnt/data/containerd. Without the mount, containerd started but had lost access to its overlay snapshot directories. All pods on that node failed to start or restart.

Root Cause

The volume at /mnt/data was not listed in /etc/fstab, so after a reboot it wasn't automatically remounted. As a result, containerd’s overlay snapshot directories (like /mnt/data/containerd/io.containerd.snapshotter.v1.overlayfs) pointed to empty or nonexistent paths. Pods using overlayfs layers based on these snapshots could not initialize. System-critical pods like the EFS CSI driver crashed with messages like "failed to stat parent".

Failure Cascade

Without functioning containerd overlayfs, pods couldn’t mount their root filesystems. This broke volume mounting, especially for PersistentVolumeClaims backed by EFS. Application pods depending on those volumes failed their readiness probes and crashed repeatedly. The kubelet was unable to recover the state because containerd's expected directory structure was invalid.

Resolution Steps

1. Ensure Persistent Mounts: Add an entry in /etc/fstab using the correct UUID to auto-mount /mnt/data on boot.

2. Clean Broken State:

   sudo systemctl stop kubelet containerd
   sudo umount -l $(mount | grep /mnt/data/containerd | awk '{print $3}')
   sudo rm -rf /mnt/data/containerd /mnt/data/containerd-state

3. Restart Core Services:

   sudo systemctl start containerd kubelet

Once containerd started cleanly and the EFS CSI driver recovered, pods came back online and the node returned to normal.

Both of these incidents highlight how Kubernetes clusters can be destabilized by subtle host-level configuration issues. Whether it’s Ubuntu's DNS stub interfering with CoreDNS or a missing mount silently breaking containerd, these are the kinds of platform engineering failures that demand tight coordination between OS setup, container runtimes, and Kubernetes orchestration.

In both cases, proactive validation during node bootstrap could have prevented downtime. Simple system checks to confirm port bindings and mount consistency—especially after OS upgrades or cloud reboots—should be standard in node lifecycle automation.

Moving forward, I've incorporated DNS stub deactivation and persistent mount verification into our golden image provisioning. It's not just about fixing what breaks, but building the node baseline that doesn't let it break the same way again.