🎉 DevOps Interview Prep Bundle is live — 1000+ Q&A across 20 topicsGet it →
All Articles

kubectl drain Not Working: Node Won't Drain — How to Fix It

Node drain stuck or failing? Getting 'cannot delete Pods' or 'PodDisruptionBudget' errors? Here are all the fixes for kubectl drain problems.

DevOpsBoysMay 28, 20264 min read
Share:Tweet

You're trying to drain a node for maintenance and kubectl just won't cooperate:

error: unable to drain node "worker-1" due to error:
cannot delete Pods not managed by ReplicationController, ReplicaSet, 
Job, DaemonSet or StatefulSet: default/standalone-pod

or

error: Cannot evict pod as it would violate the pod's disruption budget.

Here's every drain failure scenario and the exact fix.


Quick Reference: Drain Command Options

bash
# Standard drain
kubectl drain worker-1 --ignore-daemonsets
 
# Most common flags you'll need:
kubectl drain worker-1 \
  --ignore-daemonsets \        # Don't fail on DaemonSet pods (expected)
  --delete-emptydir-data \     # Allow pods with emptyDir volumes
  --grace-period=60 \          # Give pods 60s to terminate gracefully
  --timeout=300s               # Give up after 5 minutes total

Fix 1: DaemonSet Pods Blocking Drain

error: cannot delete DaemonSet-managed Pods: kube-system/kube-proxy-xxx

DaemonSet pods are expected to run on every node. Just tell drain to ignore them:

bash
kubectl drain worker-1 --ignore-daemonsets

Always use --ignore-daemonsets. DaemonSets will restart automatically when the node comes back.


Fix 2: Standalone Pods (Not Managed by Controller)

error: cannot delete Pods not managed by ReplicationController...

These are pods created directly (not via Deployment, StatefulSet, etc.). Draining would delete them permanently.

Option A: Force delete (if data loss is acceptable)

bash
kubectl drain worker-1 \
  --ignore-daemonsets \
  --force

The --force flag deletes these pods even without a controller. They won't restart.

Option B: Find and check the pod first

bash
# Find which pods are standalone
kubectl get pods --all-namespaces -o wide | grep worker-1
 
# Describe to understand what it is
kubectl describe pod PODNAME -n NAMESPACE
 
# If safe to delete, manually delete it first
kubectl delete pod PODNAME -n NAMESPACE
 
# Then drain without --force
kubectl drain worker-1 --ignore-daemonsets

Fix 3: PodDisruptionBudget Blocking Eviction

error: Cannot evict pod as it would violate the pod's disruption budget

A PDB (PodDisruptionBudget) is protecting the pod from being evicted. The PDB says "at least N pods must be running."

Find the blocking PDB:

bash
kubectl get pdb --all-namespaces
 
# Example output:
# NAMESPACE    NAME          MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
# production   api-pdb       2               N/A               0

ALLOWED DISRUPTIONS: 0 means you can't evict any pods right now — there aren't enough replicas running to satisfy the PDB.

Fix options:

Option A — Scale up the deployment first:

bash
# Scale to more replicas so PDB is satisfied
kubectl scale deployment api -n production --replicas=5
 
# Wait for pods to be ready
kubectl rollout status deployment/api -n production
 
# Now drain should work
kubectl drain worker-1 --ignore-daemonsets

Option B — Temporarily delete the PDB (risky):

bash
kubectl delete pdb api-pdb -n production
# Drain the node
kubectl drain worker-1 --ignore-daemonsets
# Recreate the PDB
kubectl apply -f pdb.yaml

Option C — Disable evictions (last resort):

bash
kubectl drain worker-1 \
  --ignore-daemonsets \
  --disable-eviction=true

This deletes pods directly instead of evicting them — bypasses PDB enforcement. Only use for emergency maintenance.


Fix 4: Pods with emptyDir Volumes

error: pods with local storage: pod-name (use --delete-emptydir-data to override)

Pods using emptyDir volumes have data that lives on the node disk. Draining would lose that data.

bash
kubectl drain worker-1 \
  --ignore-daemonsets \
  --delete-emptydir-data

emptyDir is meant to be ephemeral (caches, temp files) — data loss from this is usually acceptable. If the pod is a database using emptyDir for data, investigate before force-draining.


Fix 5: Pod Termination Timeout

Pods that take too long to gracefully shut down can block drain:

error: global timeout reached: 5m0s

Extend the timeout:

bash
kubectl drain worker-1 \
  --ignore-daemonsets \
  --timeout=600s \        # 10 minutes total
  --grace-period=120      # 2 minutes per pod

If a pod is ignoring SIGTERM and not terminating, force it:

bash
# Find the stuck pod
kubectl get pods --all-namespaces -o wide | grep worker-1
 
# Force delete it (SIGKILL)
kubectl delete pod PODNAME -n NAMESPACE --grace-period=0 --force
 
# Then re-run drain
kubectl drain worker-1 --ignore-daemonsets

Fix 6: Node Already Has a Taint/Cordon Issue

Sometimes drain commands pile up or the node gets confused:

bash
# Check node status
kubectl describe node worker-1 | grep -E "Taints|Unschedulable"
 
# Manually cordon first (marks as unschedulable)
kubectl cordon worker-1
 
# Check what pods are still running on it
kubectl get pods --all-namespaces -o wide | grep worker-1
 
# Then drain
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

After Maintenance: Uncordon the Node

Don't forget to uncordon after you're done:

bash
kubectl uncordon worker-1
 
# Verify it's back to Ready
kubectl get nodes

Complete Drain Script for Production

bash
#!/bin/bash
NODE=$1
TIMEOUT=${2:-600}
 
echo "🔒 Cordoning $NODE..."
kubectl cordon $NODE
 
echo "📊 Pods on $NODE:"
kubectl get pods --all-namespaces -o wide | grep $NODE
 
echo "🚿 Draining $NODE..."
kubectl drain $NODE \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=${TIMEOUT}s
 
if [ $? -eq 0 ]; then
  echo "✅ Node $NODE drained successfully"
  echo "Run 'kubectl uncordon $NODE' when maintenance is complete"
else
  echo "❌ Drain failed. Check pod status with:"
  echo "kubectl get pods --all-namespaces -o wide | grep $NODE"
fi

Summary

ErrorFix
DaemonSet podsAdd --ignore-daemonsets
Standalone podsAdd --force (they won't restart)
PDB violationScale up deployment or temporarily delete PDB
emptyDir dataAdd --delete-emptydir-data
TimeoutIncrease --timeout and --grace-period

CKA Tip: Draining nodes and understanding PDB behavior is tested in the CKA exam. Know the flags cold.

KodeKloud CKA Labs — practice node drain scenarios in real clusters with real PDB setups. The muscle memory from labs makes exam questions trivial.

🔧

Today I Fixed

Short real fixes from production — posted daily

Browse fixes
Newsletter

Stay ahead of the curve

Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam — just practical engineering content.

Related Articles

Comments