Prometheus Scrape Target Down — Fix
Prometheus shows your target as DOWN in the Targets page. Here's every reason a scrape target goes down and exactly how to debug and fix each one.
State: DOWN in Prometheus targets page means Prometheus can't scrape metrics from your endpoint. Here's how to diagnose and fix it.
Step 1: Check the Error Message
Go to http://prometheus:9090/targets — click on the failing target. The error message tells you exactly what's wrong:
| Error Message | Likely Cause |
|---|---|
context deadline exceeded | Target unreachable / timeout |
connection refused | Port not open / app not running |
no such host | DNS resolution failure |
Get "http://...": dial tcp: lookup ... no such host | Service name wrong |
401 Unauthorized | Auth required but not configured |
404 Not Found | Wrong metrics path |
x509: certificate... | TLS cert issue |
Case 1: Wrong Metrics Path
Default path is /metrics. If your app exposes metrics at a different path:
# In your ServiceMonitor or scrape config:
- job_name: 'my-app'
static_configs:
- targets: ['my-app:8080']
metrics_path: /actuator/prometheus # Spring Boot default
# or /stats/prometheus, /metrics/prometheus, etc.Test it manually first:
# From inside the cluster
kubectl exec -it prometheus-pod -n monitoring -- \
wget -qO- http://my-app-service.my-namespace:8080/metrics | head -20
# Or from your local machine
kubectl port-forward svc/my-app 8080:8080 -n my-namespace
curl http://localhost:8080/metricsCase 2: Network Policy Blocking Scrape
Prometheus can't reach the target because a NetworkPolicy is blocking ingress.
# Check if NetworkPolicy exists on target namespace
kubectl get networkpolicy -n my-app
# Test connectivity from prometheus pod
kubectl exec -it prometheus-0 -n monitoring -- \
wget -qO- http://my-app.my-namespace.svc.cluster.local:8080/metricsFix — allow Prometheus to scrape:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-prometheus-scrape
namespace: my-app
spec:
podSelector:
matchLabels:
app: my-app
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
ports:
- protocol: TCP
port: 8080Case 3: ServiceMonitor Not Matching
With kube-prometheus-stack, Prometheus uses ServiceMonitors. If the selector doesn't match, it won't scrape.
# Check ServiceMonitor
kubectl get servicemonitor -n my-app -o yaml
# Check if label on Service matches ServiceMonitor selector
kubectl get service my-app -n my-app --show-labelsCommon mismatch:
# ServiceMonitor looks for:
selector:
matchLabels:
app: my-app
# But Service has:
labels:
app.kubernetes.io/name: my-app # Different key!Also check: Prometheus CRD's serviceMonitorSelector — Prometheus only watches ServiceMonitors with matching labels:
kubectl get prometheus -n monitoring -o yaml | grep -A5 serviceMonitorSelectorCase 4: Port Name Mismatch
ServiceMonitor references port by name, not number.
# ServiceMonitor
endpoints:
- port: metrics # References port named "metrics"
# Service must have port with this exact name:
ports:
- name: metrics # Must match
port: 8080
targetPort: 8080Case 5: DNS Resolution Failure
# Test DNS from prometheus pod
kubectl exec -it prometheus-0 -n monitoring -- \
nslookup my-app.my-namespace.svc.cluster.local
# Check if service exists
kubectl get service my-app -n my-namespace
# Check endpoints (is there a pod behind the service?)
kubectl get endpoints my-app -n my-namespace
# If ENDPOINTS shows "<none>" → pod not running or labels don't match service selectorCase 6: TLS / HTTPS Target
# If target uses HTTPS, tell Prometheus:
- job_name: 'my-secure-app'
scheme: https
tls_config:
insecure_skip_verify: true # For self-signed certs
# Or specify CA cert:
ca_file: /etc/prometheus/certs/ca.crt
static_configs:
- targets: ['my-app:8443']Case 7: Target Pod Not Running
# Most obvious — check if pod is up
kubectl get pods -n my-namespace | grep my-app
# Check endpoints behind the service
kubectl describe endpoints my-app -n my-namespace
# "Addresses:" should list pod IPs
# Empty "Addresses" = no healthy podsDebug Flow
1. prometheus/targets → read exact error message
2. kubectl exec into prometheus → manually wget/curl the target URL
- Works? → Prometheus config/ServiceMonitor issue
- Fails? → Network/DNS/app issue
3. Check NetworkPolicy allows prometheus namespace
4. Check ServiceMonitor labels match Service labels
5. Check port name matches between ServiceMonitor and Service
6. Check pod is running and endpoints are populated
Learn Prometheus setup and troubleshooting at KodeKloud.
Today I Fixed
Short real fixes from production — posted daily
Stay ahead of the curve
Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam — just practical engineering content.
Related Articles
Prometheus High Cardinality Causing OOM — How to Find and Fix It (2026)
Prometheus is crashing with OOMKilled or running out of memory. The culprit is almost always high cardinality metrics — labels with thousands of unique values. Here's how to find which metrics are killing your Prometheus and exactly how to fix it.
Prometheus Targets Showing 'Down' — Every Cause and Fix (2026)
Your Prometheus /targets page shows red. Services are running but Prometheus can't scrape them. Here's every reason this happens — wrong port, NetworkPolicy blocks, ServiceMonitor label mismatch, auth — and exactly how to fix each one.
AI-Powered Kubernetes Anomaly Detection: Beyond Static Thresholds
Static alerts miss 40% of real incidents. Learn how AI and ML-based anomaly detection — using tools like Prometheus + ML, Dynatrace, and custom LLM runbooks — catches what thresholds can't.