Velero backup stuck in InProgress for 2 hours — node-agent OOMKilled | Today I Fixed

A scheduled Velero backup was stuck in InProgress status for over 2 hours with no movement. velero backup describe my-backup showed PVC snapshots pending but never completing.

Root cause: The Velero node-agent DaemonSet pod on one node had been OOMKilled and was in CrashLoopBackOff. Velero uses node-agent to handle PVC (file system) backups — if node-agent isn't running on a node, any PVC backup for pods scheduled on that node hangs indefinitely.

Fix:

Identify the stuck node:

bash

kubectl get pods -n velero -o wide
# Look for node-agent pods in CrashLoopBackOff or OOMKilled
kubectl describe pod -n velero <node-agent-pod> | grep -A5 "OOMKilled"

Delete the stuck backup and fix node-agent memory limits first:

bash

# Patch node-agent with higher memory limit
kubectl patch daemonset node-agent -n velero --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "1Gi"}
]'
 
# Wait for node-agent to come back healthy
kubectl rollout status daemonset/node-agent -n velero
 
# Delete the stuck backup
velero backup delete my-backup --confirm
 
# Re-run the backup
velero backup create my-backup --include-namespaces production

Monitor until completion:

bash

velero backup describe my-backup --details

Lesson: Always check kubectl get pods -n velero -o wide before chasing Velero backup issues — a dead node-agent pod is the #1 cause of hung PVC backups.