🎉 DevOps Interview Prep Bundle is live — 1000+ Q&A across 20 topicsGet it →
All Articles
monitoringobservabilitytroubleshooting

Datadog Agent Not Sending Metrics — Diagnosis and Fix Guide

Datadog dashboards show no data, hosts appear offline, or custom metrics aren't showing up. Here's how to systematically diagnose and fix Datadog agent issues on Kubernetes and VMs.

DevOpsBoysMay 15, 20264 min read
Share:Tweet

Your Datadog dashboard shows "No Data" or the host appears offline even though the agent is running. Here's how to find the actual problem fast.

Start With Agent Status

The agent status command shows everything that's working and broken:

On Kubernetes:

bash
# Get the agent pod name
kubectl get pods -n datadog -l app=datadog
 
# Run status check
kubectl exec -n datadog <datadog-agent-pod> -- agent status
 
# Short version
kubectl exec -n datadog <datadog-agent-pod> -- agent status --flare

On Linux (VM/bare metal):

bash
sudo datadog-agent status
# Or the older format:
sudo service datadog-agent status

Look for:

  • API Keys status: API key ending with XXXX: API key valid ✅
  • Collector: Running ✅
  • Any section showing ERRORS or CRITICAL ❌

Fix 1 — API Key Invalid or Missing

The most common cause of no data.

bash
# Check agent status for API key validation
kubectl exec -n datadog <pod> -- agent status | grep -A3 "API Keys"
 
# If invalid:
# API key ending with XXXX: API key invalid

Fix on Kubernetes:

bash
# Verify the secret exists and has the right key
kubectl get secret datadog-secret -n datadog -o yaml
 
# The secret should have key 'api-key'
kubectl describe secret datadog-secret -n datadog
 
# Re-create if wrong
kubectl create secret generic datadog-secret \
  --from-literal api-key=<YOUR_ACTUAL_API_KEY> \
  -n datadog \
  --dry-run=client -o yaml | kubectl apply -f -
 
# Restart the agent pods
kubectl rollout restart daemonset/datadog -n datadog

Fix on Linux:

bash
grep api_key /etc/datadog-agent/datadog.yaml
# If wrong, update it
sudo sed -i 's/api_key: .*/api_key: YOUR_CORRECT_KEY/' /etc/datadog-agent/datadog.yaml
sudo systemctl restart datadog-agent

Fix 2 — Agent Can't Reach Datadog Servers

Network connectivity issues prevent metrics from being sent.

bash
# Test connectivity from agent pod
kubectl exec -n datadog <pod> -- curl -v https://api.datadoghq.com/api/v1/validate \
  -H "DD-API-KEY: $DD_API_KEY"
 
# Test DNS resolution
kubectl exec -n datadog <pod> -- nslookup api.datadoghq.com
 
# Check agent logs for connection errors
kubectl logs -n datadog <pod> | grep -E "error|failed|timeout|connection"

Common connectivity fixes:

yaml
# If using Datadog EU site, set the site in values.yaml:
datadog:
  site: datadoghq.eu   # Default is datadoghq.com
 
# If behind a proxy:
datadog:
  env:
    - name: DD_PROXY_HTTPS
      value: "http://proxy.company.com:3128"
    - name: DD_PROXY_HTTP
      value: "http://proxy.company.com:3128"

Fix 3 — Agent Running but Metrics Missing in Dashboard

Agent appears online but specific metrics don't show up.

Check which checks are running:

bash
kubectl exec -n datadog <pod> -- agent check <check-name>
# Example:
kubectl exec -n datadog <pod> -- agent check kubelet
kubectl exec -n datadog <pod> -- agent check docker

For Kubernetes metrics specifically:

bash
# kube-state-metrics must be deployed
kubectl get pods -n datadog | grep kube-state-metrics
kubectl get pods -A | grep kube-state-metrics
 
# If missing, deploy it:
helm upgrade datadog datadog/datadog \
  --set datadog.kubeStateMetricsEnabled=true

Fix 4 — Custom Metrics Not Appearing

If your application is sending metrics via DogStatsD but they don't show up:

bash
# Verify DogStatsD is enabled
kubectl exec -n datadog <pod> -- agent status | grep -A5 "DogStatsD"
 
# Test DogStatsD from your app pod
kubectl exec -n <app-namespace> <app-pod> -- \
  nc -u -w1 <datadog-agent-service-ip> 8125 <<< "test.metric:1|c|#env:test"

Common DogStatsD configuration issues:

yaml
# In datadog helm values — ensure DogStatsD is configured correctly
datadog:
  dogstatsd:
    useHostPort: true    # Required for apps outside datadog namespace
    nonLocalTraffic: true
 
# In your app deployment — point to Datadog agent
env:
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP    # Use node IP, not cluster IP
  - name: DD_DOGSTATSD_PORT
    value: "8125"

Fix 5 — Cluster Agent Not Connecting

For Kubernetes, the Cluster Agent collects cluster-level metrics. If it's not running, you lose cluster metrics.

bash
# Check cluster agent status
kubectl get pods -n datadog | grep cluster-agent
kubectl logs -n datadog <cluster-agent-pod>
 
# Check the agent DaemonSet can reach the cluster agent
kubectl exec -n datadog <daemonset-pod> -- agent status | grep -A5 "Cluster Agent"

Fix communication between agent and cluster agent:

yaml
# In helm values, ensure cluster agent is enabled
clusterAgent:
  enabled: true
  token: <same-token-as-node-agent>
 
datadog:
  clusterAgent:
    enabled: true

Fix 6 — Logs Collection Not Working

bash
# Check if logs are collected
kubectl exec -n datadog <pod> -- agent status | grep -A10 "Logs Agent"
 
# Look for errors
kubectl logs -n datadog <pod> | grep "logs"

Enable log collection:

yaml
datadog:
  logs:
    enabled: true
    containerCollectAll: true   # Collect all container logs
 
  # Or use pod annotations for specific pods:
  # ad.datadoghq.com/app.logs: '[{"source":"python","service":"my-app"}]'

Quick Diagnosis Flowchart

Agent running? (kubectl get pods -n datadog)
  │
  No → Check DaemonSet tolerations, node selector
  │
  Yes → agent status
         │
         API key invalid? → Fix secret, restart agent
         │
         API key valid, no metrics → Check connectivity (curl api.datadoghq.com)
         │
         Connectivity OK → Check specific check (agent check <name>)
         │
         Check failing → Missing dependency (kube-state-metrics, etc.)

Useful Datadog Agent Commands

bash
# Full status (most useful)
kubectl exec -n datadog <pod> -- agent status
 
# Test a specific integration check
kubectl exec -n datadog <pod> -- agent check nginx
 
# Send a test event
kubectl exec -n datadog <pod> -- agent flare
 
# Show configured checks
kubectl exec -n datadog <pod> -- agent configcheck
 
# Diagnose connectivity
kubectl exec -n datadog <pod> -- agent diagnose

For observability and monitoring hands-on labs including Datadog and Prometheus, KodeKloud has DevOps monitoring courses with real scenario-based exercises.

Free Tools
Free Tool

K8s Resource Calculator

Calculate CPU & memory requests/limits for your pods. Never get OOMKilled again.

Free Tool

YAML Validator

Paste your Kubernetes manifests, Helm values, or Terraform config — catch errors instantly.

Back to all articles
Share:Tweet
Newsletter

Stay ahead of the curve

Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam — just practical engineering content.

Related Articles

Comments