Updated a Kubernetes Secret but pod was still reading the old value

Rotated a database password in a Kubernetes Secret, verified with kubectl get secret that the new value was there — but the running pod was still connecting with the old password and failing auth.

Root cause: Kubernetes does not automatically restart pods when a Secret changes. When Secrets are injected as environment variables, the values are baked in at pod start time and never updated until the pod restarts. (Secrets mounted as volumes do eventually refresh, but it can take 60–90 seconds and depends on kubelet sync period.)

Fix:

After updating the Secret, force a rolling restart:

bash

# Update the secret
kubectl create secret generic db-secret \
  --from-literal=DB_PASSWORD='newpassword123' \
  --dry-run=client -o yaml | kubectl apply -f -
 
# Restart the deployment to pick up new values
kubectl rollout restart deployment my-app -n production
 
# Verify rollout completes
kubectl rollout status deployment my-app -n production

If you want near-instant secret refresh without restarts, use volume mounts instead of envFrom, and read the file at runtime:

yaml

volumeMounts:
  - name: secret-vol
    mountPath: /etc/secrets
    readOnly: true

Lesson: kubectl rollout restart is your best friend after any Secret or ConfigMap change — never assume the pod picked up the new value automatically.