Grafana showing 'No data' after moving Prometheus to a new namespace

After migrating the Prometheus stack from the monitoring namespace to observability, every Grafana dashboard went blank — showing "No data" for all panels. The Prometheus UI itself was fine and scraping targets normally.

Root cause: Grafana's Prometheus datasource URL was hardcoded to the old internal service DNS: http://prometheus.monitoring:9090. After the namespace change, the service moved to http://prometheus.observability:9090. Grafana couldn't reach Prometheus at the old address, so all queries returned empty.

Fix:

Go to Grafana → Settings → Data Sources → Prometheus, update the URL:

# Old (broken)
http://prometheus.monitoring:9090

# New (correct)
http://prometheus.observability:9090

Or if managing datasources as code via ConfigMap:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: observability
data:
  prometheus.yaml: |
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus.observability:9090
        isDefault: true

Apply and restart Grafana:

bash

kubectl apply -f grafana-datasources.yaml
kubectl rollout restart deployment grafana -n observability

Then hit Save & Test in the Grafana UI — you should see "Data source is working".

Lesson: Kubernetes internal DNS is namespace-scoped — whenever you move a service to a new namespace, update every hardcoded service URL that points to it.