kubectl: Error from server - connection refused | Today I Fixed

Was trying to run kubectl get pods and got this:

The connection to the server 10.0.1.45:6443 was refused - did you specify the right host or port?

Spent 25 minutes debugging. Turned out the API server pod on the control plane node had crashed because the node ran out of disk space. The /var/log directory was full.

What I did:

Checked the control plane node first:

bash

df -h  # disk was 100% on /var/lib

Cleared old container logs:

bash

journalctl --vacuum-time=3d
crictl rmi --prune  # remove unused container images

After freeing ~15GB, the API server came back up automatically within 2 minutes. kubectl started working again.

Lesson: Set up disk usage alerts on control plane nodes. I now alert at 80% disk usage. Also rotate logs with a logrotate config.