Prometheus alert not firing even though metric was above threshold

The problem: CPU alert was configured to fire at 80% but Prometheus wasn't sending any alerts even though node_cpu_seconds_total was clearly showing high usage in Grafana.

The fix:

bash

# Check Prometheus alerts page: http://prometheus:9090/alerts
# Status showed: "inactive" — rule was never evaluating to true
 
# The issue was in the PromQL expression:
# WRONG — this computes instant rate, often returns 0 or incorrect:
alert: HighCPU
expr: node_cpu_seconds_total > 0.8
 
# CORRECT — rate over time window, minus idle, as percentage:
alert: HighCPU
expr: (1 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100 > 80
for: 5m

bash

# Also check AlertManager is receiving alerts:
kubectl port-forward svc/alertmanager 9093:9093 -n monitoring
# http://localhost:9093 → check if alerts appear here
 
# Test the PromQL expression directly in Prometheus UI:
# http://prometheus:9090/graph → paste expression → check if it returns values

Why it happens: node_cpu_seconds_total is a counter (always increasing), not a percentage. Comparing it directly to 0.8 almost never returns true because the raw value is billions of seconds accumulated since boot. You must use rate() to get per-second change, subtract the idle mode, and convert to percentage. Always test PromQL expressions in the Prometheus UI before putting them in alert rules.