docker build kept failing halfway through with:
ERROR: failed to solve: write /var/lib/docker/tmp/...: no space left on device
Even though df -h showed 40GB free on /. Turned out Docker's overlay filesystem was consuming all of /var/lib/docker which was on a separate smaller partition.
What fixed it:
# See how much Docker is using
docker system df
# Prune everything unused (stopped containers, dangling images, unused volumes)
docker system prune -a --volumes
# Specifically remove dangling images (saves most space)
docker image prune -aFreed 23GB of old build cache and unused images. Build succeeded immediately after.
Prevention: Added a weekly cron to prune on CI runners:
# /etc/cron.weekly/docker-prune
docker system prune -f --filter "until=168h"The --filter "until=168h" only removes things older than 7 days, so fresh cache layers survive.