Kubernetes HPA Not Scaling on Custom Metrics Fix

You added a custom metric to your HPA — SQS queue depth, HTTP request rate from Prometheus, whatever — and the HPA stubbornly sits at one replica. kubectl get hpa shows TARGETS: <unknown>/100. CPU autoscaling works fine, but custom metrics never trigger. Here is exactly what is happening and how to fix it.

How Custom Metrics Autoscaling Works

Kubernetes HPA v2 can scale on three sources:

1. Resource metrics — CPU, memory (built-in, always works)
2. Custom metrics — metrics from your apps, exposed via the custom metrics API
3. External metrics — metrics from outside the cluster (SQS depth, Pub/Sub lag)

For #2 and #3 to work, you need a metrics adapter running in your cluster that implements the custom.metrics.k8s.io and external.metrics.k8s.io API groups. The two main options are prometheus-adapter and KEDA.

Without the adapter, HPA cannot read the metric and shows <unknown>.

Step 1: Verify the Custom Metrics API Exists

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | python3 -m json.tool | head -20

If you get:

Error from server (NotFound): the server could not find the requested resource

No adapter is installed. That is your root cause. Jump to the installation section.

If you get a JSON response, the API exists. Now check if your specific metric is registered:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests_per_second"

Expected output when working:

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "describedObject": { "kind": "Pod", "name": "my-app-7d9f6b-xkp2m" },
      "metricName": "http_requests_per_second",
      "value": "42"
    }
  ]
}

If it returns empty items: [] or 404, the metric name in your HPA does not match what the adapter is exposing.

Step 2: Check Your HPA Spec

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 100

Common mistakes:

• type: Pods vs type: Object — use Pods for per-pod metrics, Object for cluster-level metrics
• Metric name must exactly match what the adapter exposes — no underscores vs dashes mismatch
• Wrong namespace — the HPA and the metric must be in the same namespace

Step 3: Install and Configure prometheus-adapter

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
 
helm install prometheus-adapter prometheus-community/prometheus-adapter \
  --namespace monitoring \
  --set prometheus.url=http://prometheus-operated.monitoring.svc \
  --set prometheus.port=9090

The critical part is the ConfigMap that tells the adapter how to translate Prometheus metrics into Kubernetes custom metrics:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-adapter
  namespace: monitoring
data:
  config.yaml: |
    rules:
    - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: "^(.*)_total$"
        as: "${1}_per_second"
      metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[2m])'

This rule takes http_requests_total from Prometheus and exposes it as http_requests_per_second to the HPA — at 2-minute rate.

After applying, restart the adapter:

kubectl rollout restart deployment/prometheus-adapter -n monitoring

Verify after 60 seconds:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests_per_second"

Step 4: Debug HPA Events

kubectl describe hpa my-app-hpa -n default

Look at the Events section:

Warning  FailedGetScale  unable to get metrics for resource pods: unable to fetch metrics from custom metrics API
Warning  FailedComputeMetricsReplicas  invalid metrics (1 invalid out of 1): failed to get pods metric value: unable to get metrics for resource pods

The first error means the metric name is wrong or the adapter is not running. The second means the metric returned zero values.

KEDA: The Better Alternative

If prometheus-adapter configuration feels complex, KEDA (Kubernetes Event Driven Autoscaler) is a much simpler drop-in that supports 60+ scalers out of the box — Prometheus, SQS, Kafka, Redis, Cron, and more.

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

Replace your HPA with a KEDA ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-app-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: my-app
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-operated.monitoring.svc:9090
      metricName: http_requests_per_second
      query: rate(http_requests_total{namespace="default"}[2m])
      threshold: "100"

KEDA creates and manages the HPA for you. No adapter ConfigMap rules, no API registration dance.

Check ScaledObject status:

kubectl get scaledobject my-app-scaler -n default
# NAME             SCALETARGETKIND   SCALETARGETNAME   MIN   MAX   READY   ACTIVE
# my-app-scaler    Deployment        my-app            1     10    True    True

Quick Checklist

• Custom metrics API exists: kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
• Metric name in HPA exactly matches adapter output
• prometheus-adapter ConfigMap rules cover your Prometheus series
• Pod labels match the LabelMatchers in the adapter rule
• If still broken: switch to KEDA ScaledObject — simpler and more reliable

Custom metrics autoscaling has more moving parts than CPU autoscaling, but once the adapter is configured correctly (or you switch to KEDA), it just works.