🎉 DevOps Interview Prep Bundle is live — 1000+ Q&A across 20 topicsGet it →
All Articles

Kubernetes CrashLoopBackOff: Complete Fix Guide for Every Cause

CrashLoopBackOff in Kubernetes? This guide covers every cause — bad command, OOMKill, readiness probe killing container, config missing, dependency not ready — with exact kubectl commands to find and fix each one.

Shubham4 min read
Share:Tweet

CrashLoopBackOff means Kubernetes is starting your container, it crashes, Kubernetes restarts it, it crashes again — and the wait between restarts doubles each time (10s → 20s → 40s → 160s → 300s max).

First: Find Why It Is Crashing

bash
# See current state
kubectl get pod myapp-xyz -n production
# NAME         READY   STATUS             RESTARTS   AGE
# myapp-xyz    0/1     CrashLoopBackOff   5          10m
 
# Get logs from the crashed container
kubectl logs myapp-xyz -n production
 
# If current container has no logs, get from the previous run
kubectl logs myapp-xyz -n production --previous
 
# Check events for the pod
kubectl describe pod myapp-xyz -n production | grep -A 30 "Events:"

The --previous flag is the most important one — it shows logs from the container run that crashed, before Kubernetes restarted it.

Cause 1: Bad or Missing Entrypoint/Command

bash
# In describe output:
# Last State: Terminated
# Reason: Error
# Exit Code: 127    ← "command not found"
# or
# Exit Code: 1      ← application startup error

Exit code 127 = the command in CMD or command: does not exist in the image.

bash
# What is the container actually running?
kubectl get pod myapp-xyz -n production -o jsonpath='{.spec.containers[0].command}'
kubectl get pod myapp-xyz -n production -o jsonpath='{.spec.containers[0].args}'
 
# Test the image locally
docker run --rm myapp/api:1.2.3 which python3
docker run --rm myapp/api:1.2.3 ls /app/

Fix: Match the command to what actually exists in the image:

yaml
containers:
- name: api
  image: myapp/api:1.2.3
  command: ["python3"]      # Must exist in image
  args: ["-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Cause 2: OOMKilled — Out of Memory

bash
# In describe output:
# Last State: Terminated
# Reason: OOMKilled    ← This is the killer
# Exit Code: 137
 
# Check current memory usage
kubectl top pod myapp-xyz -n production

Exit code 137 = container killed by kernel (OOM). Your memory limit is too low.

bash
# Check current limits
kubectl get pod myapp-xyz -n production -o jsonpath='{.spec.containers[0].resources}'
# {"limits":{"memory":"256Mi"},"requests":{"memory":"128Mi"}}

Fix — increase memory limit:

bash
kubectl patch deployment myapp -n production -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"api","resources":{"limits":{"memory":"512Mi"}}}]}}}}'

Or edit the deployment:

yaml
resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"    # Start 2x what you think you need
    cpu: "500m"

Cause 3: Missing Environment Variable or Config

The app starts but crashes because DATABASE_URL or another env var is empty.

bash
# Look for this in logs:
kubectl logs myapp-xyz -n production --previous | grep -i "env\|config\|missing\|required\|undefined\|null"
 
# Common patterns:
# "KeyError: 'DATABASE_URL'"
# "Configuration error: REDIS_URL is not set"
# "Cannot read property of undefined"
 
# Check what env vars are actually set on the pod
kubectl exec myapp-xyz -n production -- env | sort

Fix — ensure the Secret and ConfigMap exist and are mounted:

bash
# Verify the secret exists
kubectl get secret myapp-secrets -n production
 
# Verify it has the expected keys
kubectl get secret myapp-secrets -n production -o jsonpath='{.data}' | base64 -d
 
# Check the deployment references it correctly
kubectl get deployment myapp -n production -o yaml | grep -A 20 env

Deployment should look like:

yaml
env:
- name: DATABASE_URL
  valueFrom:
    secretKeyRef:
      name: myapp-secrets
      key: database-url    # Must match exact key name in secret
      optional: false      # Will fail loudly if missing

Cause 4: Dependency Not Ready (DB, Redis)

The app crashes on startup because it cannot connect to its database.

bash
# Look for connection errors in logs:
kubectl logs myapp-xyz -n production --previous | grep -i "connect\|refused\|timeout"
# "Connection refused: db-service:5432"
# "ECONNREFUSED 10.0.0.5:6379"

Fix — add init containers to wait for dependencies:

yaml
initContainers:
- name: wait-for-db
  image: busybox:1.36
  command: ['sh', '-c', 'until nc -z db-service 5432; do echo waiting for db; sleep 2; done']
- name: wait-for-redis
  image: busybox:1.36
  command: ['sh', '-c', 'until nc -z redis-service 6379; do echo waiting for redis; sleep 2; done']
containers:
- name: api
  # ...

Cause 5: Liveness Probe Killing Healthy Container

A misconfigured liveness probe kills the container before it finishes starting.

bash
# In describe output:
# Warning  Unhealthy  Liveness probe failed: HTTP probe failed with statuscode: 503
# Normal   Killing    Container api failed liveness probe, will be restarted

The container is starting fine but the liveness probe fires before the app is ready.

Fix — add initialDelaySeconds:

yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 60    # Wait 60s before first check
  periodSeconds: 30
  failureThreshold: 3
  timeoutSeconds: 10

Or better: use separate startup probe (Kubernetes 1.18+):

yaml
startupProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 30       # Allow 30 × 10s = 5 minutes to start
  periodSeconds: 10
 
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 30          # Only runs after startup probe passes

Quick Fix Checklist

bash
# Step 1: Get crash reason
kubectl describe pod POD_NAME -n NS | grep "Last State:" -A 5
# Look for: OOMKilled, Error, Completed
 
# Step 2: Get crash logs
kubectl logs POD_NAME -n NS --previous 2>&1 | tail -50
 
# Step 3: Check events
kubectl describe pod POD_NAME -n NS | grep Events: -A 20
 
# Step 4: Verify env vars
kubectl exec POD_NAME -n NS -- env | grep -i "url\|host\|key\|secret"
 
# Step 5: Test connectivity from pod
kubectl exec POD_NAME -n NS -- nc -zv db-service 5432

More Kubernetes troubleshooting? Read our Kubernetes OOMKilled fix and Kubernetes HPA not scaling fix.

🔧

Today I Fixed

Short real fixes from production — posted daily

Browse fixes
Newsletter

Stay ahead of the curve

Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam — just practical engineering content.

Related Articles

Comments