Server disk at 100%, application failing to write logs — finding and clearing space fast

Application pods started crashing with No space left on device. df -h confirmed / was at 100%. SSH was still working but nothing else was writing.

Root cause: A combination of Docker image accumulation (old images from 30+ deploys) and a runaway log file under /var/log that had grown to 18 GB with no rotation configured.

Fix:

Step 1 — find what's consuming space:

du -sh /* 2>/dev/null | sort -rh | head -20

Step 2 — drill into the biggest offender (was /var):

du -sh /var/* 2>/dev/null | sort -rh | head -10
find /var/log -name "*.log" -size +100M

Step 3 — truncate the large log file without breaking open file handles:

truncate -s 0 /var/log/myapp/app.log

Step 4 — clear unused Docker objects:

docker system prune -a --volumes

This recovered 22 GB. Then set up log rotation to prevent recurrence:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
}

Lesson: Never delete a log file with rm while the process has it open — use truncate -s 0 instead, or the disk space won't actually be freed until the process restarts.