🎉 DevOps Interview Prep Bundle is live — 1000+ Q&A across 20 topicsGet it →
All Articles

kubectl drain Not Working: Node Won't Drain — How to Fix It

Node drain stuck or failing? Getting 'cannot delete Pods' or 'PodDisruptionBudget' errors? Here are all the fixes for kubectl drain problems.

DevOpsBoys4 min read
Share:Tweet

You're trying to drain a node for maintenance and kubectl just won't cooperate:

error: unable to drain node "worker-1" due to error:
cannot delete Pods not managed by ReplicationController, ReplicaSet, 
Job, DaemonSet or StatefulSet: default/standalone-pod

or

error: Cannot evict pod as it would violate the pod's disruption budget.

Here's every drain failure scenario and the exact fix.


Quick Reference: Drain Command Options

bash
# Standard drain
kubectl drain worker-1 --ignore-daemonsets
 
# Most common flags you'll need:
kubectl drain worker-1 \
  --ignore-daemonsets \        # Don't fail on DaemonSet pods (expected)
  --delete-emptydir-data \     # Allow pods with emptyDir volumes
  --grace-period=60 \          # Give pods 60s to terminate gracefully
  --timeout=300s               # Give up after 5 minutes total

Fix 1: DaemonSet Pods Blocking Drain

error: cannot delete DaemonSet-managed Pods: kube-system/kube-proxy-xxx

DaemonSet pods are expected to run on every node. Just tell drain to ignore them:

bash
kubectl drain worker-1 --ignore-daemonsets

Always use --ignore-daemonsets. DaemonSets will restart automatically when the node comes back.


Fix 2: Standalone Pods (Not Managed by Controller)

error: cannot delete Pods not managed by ReplicationController...

These are pods created directly (not via Deployment, StatefulSet, etc.). Draining would delete them permanently.

Option A: Force delete (if data loss is acceptable)

bash
kubectl drain worker-1 \
  --ignore-daemonsets \
  --force

The --force flag deletes these pods even without a controller. They won't restart.

Option B: Find and check the pod first

bash
# Find which pods are standalone
kubectl get pods --all-namespaces -o wide | grep worker-1
 
# Describe to understand what it is
kubectl describe pod PODNAME -n NAMESPACE
 
# If safe to delete, manually delete it first
kubectl delete pod PODNAME -n NAMESPACE
 
# Then drain without --force
kubectl drain worker-1 --ignore-daemonsets

Fix 3: PodDisruptionBudget Blocking Eviction

error: Cannot evict pod as it would violate the pod's disruption budget

A PDB (PodDisruptionBudget) is protecting the pod from being evicted. The PDB says "at least N pods must be running."

Find the blocking PDB:

bash
kubectl get pdb --all-namespaces
 
# Example output:
# NAMESPACE    NAME          MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
# production   api-pdb       2               N/A               0

ALLOWED DISRUPTIONS: 0 means you can't evict any pods right now — there aren't enough replicas running to satisfy the PDB.

Fix options:

Option A — Scale up the deployment first:

bash
# Scale to more replicas so PDB is satisfied
kubectl scale deployment api -n production --replicas=5
 
# Wait for pods to be ready
kubectl rollout status deployment/api -n production
 
# Now drain should work
kubectl drain worker-1 --ignore-daemonsets

Option B — Temporarily delete the PDB (risky):

bash
kubectl delete pdb api-pdb -n production
# Drain the node
kubectl drain worker-1 --ignore-daemonsets
# Recreate the PDB
kubectl apply -f pdb.yaml

Option C — Disable evictions (last resort):

bash
kubectl drain worker-1 \
  --ignore-daemonsets \
  --disable-eviction=true

This deletes pods directly instead of evicting them — bypasses PDB enforcement. Only use for emergency maintenance.


Fix 4: Pods with emptyDir Volumes

error: pods with local storage: pod-name (use --delete-emptydir-data to override)

Pods using emptyDir volumes have data that lives on the node disk. Draining would lose that data.

bash
kubectl drain worker-1 \
  --ignore-daemonsets \
  --delete-emptydir-data

emptyDir is meant to be ephemeral (caches, temp files) — data loss from this is usually acceptable. If the pod is a database using emptyDir for data, investigate before force-draining.


Fix 5: Pod Termination Timeout

Pods that take too long to gracefully shut down can block drain:

error: global timeout reached: 5m0s

Extend the timeout:

bash
kubectl drain worker-1 \
  --ignore-daemonsets \
  --timeout=600s \        # 10 minutes total
  --grace-period=120      # 2 minutes per pod

If a pod is ignoring SIGTERM and not terminating, force it:

bash
# Find the stuck pod
kubectl get pods --all-namespaces -o wide | grep worker-1
 
# Force delete it (SIGKILL)
kubectl delete pod PODNAME -n NAMESPACE --grace-period=0 --force
 
# Then re-run drain
kubectl drain worker-1 --ignore-daemonsets

Fix 6: Node Already Has a Taint/Cordon Issue

Sometimes drain commands pile up or the node gets confused:

bash
# Check node status
kubectl describe node worker-1 | grep -E "Taints|Unschedulable"
 
# Manually cordon first (marks as unschedulable)
kubectl cordon worker-1
 
# Check what pods are still running on it
kubectl get pods --all-namespaces -o wide | grep worker-1
 
# Then drain
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

After Maintenance: Uncordon the Node

Don't forget to uncordon after you're done:

bash
kubectl uncordon worker-1
 
# Verify it's back to Ready
kubectl get nodes

Complete Drain Script for Production

bash
#!/bin/bash
NODE=$1
TIMEOUT=${2:-600}
 
echo "🔒 Cordoning $NODE..."
kubectl cordon $NODE
 
echo "📊 Pods on $NODE:"
kubectl get pods --all-namespaces -o wide | grep $NODE
 
echo "🚿 Draining $NODE..."
kubectl drain $NODE \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=${TIMEOUT}s
 
if [ $? -eq 0 ]; then
  echo "✅ Node $NODE drained successfully"
  echo "Run 'kubectl uncordon $NODE' when maintenance is complete"
else
  echo "❌ Drain failed. Check pod status with:"
  echo "kubectl get pods --all-namespaces -o wide | grep $NODE"
fi

Summary

ErrorFix
DaemonSet podsAdd --ignore-daemonsets
Standalone podsAdd --force (they won't restart)
PDB violationScale up deployment or temporarily delete PDB
emptyDir dataAdd --delete-emptydir-data
TimeoutIncrease --timeout and --grace-period

CKA Tip: Draining nodes and understanding PDB behavior is tested in the CKA exam. Know the flags cold.

KodeKloud CKA Labs — practice node drain scenarios in real clusters with real PDB setups. The muscle memory from labs makes exam questions trivial.

🔧

Today I Fixed

Short real fixes from production — posted daily

Browse fixes
Newsletter

Stay ahead of the curve

Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam — just practical engineering content.

Related Articles

Comments