Kubernetes Pods Getting Evicted Due to Disk Pressure — Here's the Fix

Waking up to hundreds of evicted pods is one of those production moments that nobody forgets. The error message is usually something like:

The node was low on resource: ephemeral-storage.
Threshold quantity: 10%, available: 8%.

Or you see nodes with DiskPressure=True condition, and pods keep getting killed faster than they restart.

Here's what's actually happening and how to fix it.

Why Kubernetes Evicts Pods for Disk Pressure

Kubernetes monitors disk usage on each node. When available disk space drops below a threshold (default: 10% or 100Mi, whichever comes first), the kubelet triggers eviction — it kills pods to reclaim disk space.

The eviction is aggressive by design. Kubernetes prioritizes node stability over individual pod availability. A node that runs out of disk space completely can crash, which is worse than a few evicted pods.

The disk space being consumed is usually one of:

• Container logs accumulating without rotation
• Large container images filling /var/lib/docker or /var/lib/containerd
• Application temp files written to /tmp inside containers (counts as ephemeral storage)
• Crashed containers leaving behind layers on disk

Step 1: Find What's Eating Disk Space

SSH into the affected node and run:

df -h

Look at which filesystem is nearly full. Usually it's the root filesystem or a dedicated disk for container storage.

Then find the big directories:

du -sh /var/lib/containerd/*   # for containerd
du -sh /var/lib/docker/*       # for Docker runtime
du -sh /var/log/pods/*         # pod logs
du -sh /tmp/*

In most cases I've seen, the culprit is either:

1. /var/log/pods/ — logs from containers that had a lot of output
2. /var/lib/containerd/ — old image layers and stopped container data

Step 2: Clean Up Immediately

To stop the evictions, you need to free disk space fast.

Clean up stopped containers and unused images:

# For containerd
crictl rmi --prune
 
# For Docker
docker system prune -af --volumes

Clean up old pod logs:

find /var/log/pods -name "*.log" -mtime +3 -delete
find /var/log/containers -name "*.log" -mtime +3 -delete

After cleanup, check if the node DiskPressure condition clears:

kubectl describe node <node-name> | grep -A5 Conditions

The kubelet checks disk every 10 seconds by default, so it should clear within a minute.

Step 3: Find Which Pod Is Generating the Most Disk Usage

If cleanup keeps being necessary, something is writing too much data. Find it:

# Check ephemeral storage usage per pod
kubectl get pods -A -o json | jq -r '
  .items[] |
  "\(.metadata.namespace)/\(.metadata.name): \(.status.containerStatuses[0]?.state // "unknown")"
' 
 
# Better: use kubectl top if metrics-server is running
kubectl top pods -A --sort-by=memory

For ephemeral storage specifically:

kubectl describe node <node-name> | grep -A30 "Allocated resources"

Step 4: Add Ephemeral Storage Limits to Your Pods

This is the fix that prevents recurrence. Without ephemeral storage limits, a single pod with runaway logging or temp file creation can fill a node.

Add resource limits to your pod spec:

resources:
  requests:
    ephemeral-storage: "500Mi"
  limits:
    ephemeral-storage: "2Gi"

When a pod exceeds its ephemeral storage limit, Kubernetes evicts only that pod — not everything on the node. This is much better than the node-level disk pressure eviction.

Step 5: Set Up Log Rotation

Container runtimes handle log rotation, but you might need to configure it explicitly.

For containerd, check /etc/containerd/config.toml:

[plugins."io.containerd.grpc.v1.cri".containerd]
  # Log rotation settings
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

For the kubelet, configure log rotation in /var/lib/kubelet/config.yaml:

containerLogMaxSize: "50Mi"
containerLogMaxFiles: 3

These settings rotate container logs when they hit 50MB and keep only 3 rotated files per container. Restart the kubelet after changing this.

Step 6: Expand Node Disk or Add Nodes

If your cluster is legitimately running out of disk due to growing workloads, you need more capacity:

Expand EBS volume (AWS EKS):

# Modify the EBS volume size in your node group
aws ec2 modify-volume --volume-id vol-xxx --size 200
 
# Extend the filesystem after resize
sudo growpart /dev/nvme0n1 1
sudo resize2fs /dev/nvme0n1p1

Or adjust your cluster autoscaler to add nodes before disk gets full — set target utilization lower so new nodes join before you hit disk pressure.

Quick Reference: Eviction vs OOMKilled vs CrashLoopBackOff

The Permanent Fix

The sequence that actually prevents this long-term:

1. Set containerLogMaxSize: 50Mi in kubelet config
2. Add ephemeral-storage limits to all workload specs
3. Set up a CronJob to prune unused container images weekly
4. Add a Prometheus alert for node disk usage above 75%

Don't wait for evictions to happen before you notice — by then pods are already dead.

Dealing with other resource issues? Check out our guides on fixing OOMKilled pods and Kubernetes resource requests and limits explained.