Kubernetes pods stuck in Pending — 0/3 nodes are available: 3 Insufficient cpu

Deployed a new service and all pods stayed in Pending state. kubectl describe pod showed the event: 0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found.

Root cause: The Deployment had resources.requests.cpu: 2000m (2 full CPUs per pod), but each node only had ~1.5 CPU allocatable after system pods reserved their share. Kubernetes couldn't schedule the pod anywhere because no single node had 2 free CPUs.

Fix:

Check actual allocatable resources on nodes:

kubectl describe nodes | grep -A 5 "Allocatable:"

Check what's already consuming resources:

kubectl describe nodes | grep -A 10 "Allocated resources:"

Fix the resource requests to match realistic usage:

resources:
  requests:
    cpu: "500m"      # was 2000m — way too high for this app
    memory: "256Mi"
  limits:
    cpu: "1000m"
    memory: "512Mi"

Apply and verify:

kubectl apply -f deployment.yaml
kubectl get pods -w

Also check if cluster autoscaler or Karpenter is configured — if it is and pods are still Pending, the node group may have hit its max size limit:

kubectl get nodes
kubectl describe configmap cluster-autoscaler-status -n kube-system

Lesson: Always set CPU requests based on actual profiled usage, not perceived need — oversized requests block scheduling even when nodes have available capacity in practice.