Kubernetes nodes stuck NotReady after cluster upgrade

Upgraded EKS from 1.28 to 1.29. Two worker nodes went NotReady and stayed there for 30+ minutes.

bash

kubectl get nodes
# NAME                        STATUS     ROLES    AGE
# ip-10-0-1-22.ec2.internal   NotReady   <none>   18m
# ip-10-0-1-45.ec2.internal   NotReady   <none>   18m

Debugging:

bash

kubectl describe node ip-10-0-1-22.ec2.internal
# Conditions:
#   Ready   False   KubeletNotReady   container runtime network not ready:
#            NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: 
#            cni plugin not initialized

The CNI (Calico in our case) wasn't initialized on the upgraded nodes.

What happened: Calico's node DaemonSet was running on the old version that wasn't compatible with Kubernetes 1.29. When the nodes upgraded their kubelet, it expected the new CNI version, but the Calico pod was still on the old version and crashed.

bash

kubectl get pods -n kube-system -l k8s-app=calico-node
# calico-node-abc  0/1  CrashLoopBackOff
kubectl logs -n kube-system calico-node-abc
# Error: incompatible Kubernetes version

Fix:

Upgraded Calico first (should have done this before the node upgrade):

bash

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml

The Calico DaemonSet rolled out new pods. Nodes transitioned to Ready within 5 minutes.

The right order for EKS upgrades:

Upgrade the control plane first
Update add-ons (CoreDNS, kube-proxy, VPC CNI, Calico) BEFORE upgrading node groups
Upgrade managed node groups
Upgrade self-managed node groups

Lesson: Always upgrade CNI and other add-ons BEFORE upgrading nodes. The add-ons often have version compatibility requirements with specific Kubernetes versions.