AWS ALB Showing Unhealthy Targets — How to Fix It

Your ALB target group shows all targets as "unhealthy." Your application is running fine — you can curl it directly from the EC2 instance or pod. But the load balancer refuses to route traffic, and users are getting 502 or 503 errors.

This is one of the most common AWS networking issues, and it's almost always a configuration mismatch between what the ALB expects and what your application provides. Let's fix it.

How ALB Health Checks Work

The ALB sends HTTP requests to your targets at regular intervals. If a target doesn't respond with the expected status code within the timeout, it's marked unhealthy after a threshold number of failures.

ALB → Health check request (GET /health) → Target (EC2/Pod/IP)
     ← Expected: 200 OK within 5s
     ← Actual: timeout / 404 / connection refused
     → Target marked UNHEALTHY

The health check configuration includes:

• Protocol: HTTP or HTTPS
• Port: which port to check (traffic port or override)
• Path: URL path to request (default: /)
• Healthy threshold: consecutive successes needed (default: 5)
• Unhealthy threshold: consecutive failures to mark unhealthy (default: 2)
• Timeout: seconds to wait for response (default: 5)
• Interval: seconds between checks (default: 30)
• Success codes: expected HTTP status codes (default: 200)

Problem 1: Health Check Path Returns 404

The Symptom

Target group health check shows:

Health check failed with error: "HTTP 404: Not Found"

Why It Happens

The ALB is checking a path that doesn't exist in your application. The default health check path is /, but many applications:

• Serve their health endpoint at /health, /healthz, or /api/health
• Return a redirect (301/302) from / to /login or /dashboard
• Require authentication on /

The Fix

Option 1: Update the health check path in the target group

aws elbv2 modify-target-group \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/my-tg/abc123 \
  --health-check-path /health \
  --health-check-protocol HTTP

Option 2: Accept more status codes

If your app redirects from /, accept 301/302 as healthy:

aws elbv2 modify-target-group \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/my-tg/abc123 \
  --matcher '{"HttpCode":"200,301,302"}'

Option 3: Add a health endpoint to your app

// Express.js
app.get('/health', (req, res) => {
  res.status(200).json({ status: 'ok' });
});

# FastAPI
@app.get("/health")
def health():
    return {"status": "ok"}

Problem 2: Security Group Blocking Health Checks

The Symptom

Health checks timeout. You can access the app from the instance itself but not from the ALB.

Why It Happens

The EC2 instance or EKS node security group doesn't allow inbound traffic from the ALB's security group.

The Fix

Step 1: Find the ALB's security group

aws elbv2 describe-load-balancers --names my-alb \
  --query 'LoadBalancers[0].SecurityGroups'

Step 2: Add an inbound rule to the target's security group

# Allow traffic from ALB security group on the health check port
aws ec2 authorize-security-group-ingress \
  --group-id sg-target-instance \
  --protocol tcp \
  --port 8080 \
  --source-group sg-alb-security-group

For EKS with AWS Load Balancer Controller, the annotation handles this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-target-type: ip
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080

The ALB controller automatically configures security group rules when target-type: ip is used.

Problem 3: Health Check Port Mismatch

The Symptom

Your app runs on port 8080, but the health check is hitting port 80.

Why It Happens

The target group was created with port 80, or the health check port override is wrong.

The Fix

# Check current health check configuration
aws elbv2 describe-target-groups \
  --target-group-arns arn:aws:elasticloadbalancing:... \
  --query 'TargetGroups[0].{Port:Port,HealthCheckPort:HealthCheckPort,HealthCheckPath:HealthCheckPath}'
 
# Fix: set health check to use traffic port
aws elbv2 modify-target-group \
  --target-group-arn arn:aws:elasticloadbalancing:... \
  --health-check-port traffic-port

traffic-port means "use the same port as the target registration" — this is usually what you want.

Problem 4: App Slow to Start (Health Check Timeout)

The Symptom

Targets start as unhealthy, then eventually become healthy after several minutes. During deployments, you get downtime because new targets are unhealthy while old ones are being drained.

Why It Happens

Your application takes 30-60 seconds to start (JVM warmup, database migrations, cache priming), but the health check starts immediately and the timeout is only 5 seconds.

The Fix

Increase the health check grace period and adjust thresholds:

aws elbv2 modify-target-group \
  --target-group-arn arn:aws:elasticloadbalancing:... \
  --health-check-interval-seconds 15 \
  --health-check-timeout-seconds 10 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 5

This gives your app: 5 failures × 15 seconds = 75 seconds to become healthy.

For Kubernetes, use a startup probe alongside the ALB health check:

spec:
  containers:
    - name: app
      startupProbe:
        httpGet:
          path: /health
          port: 8080
        initialDelaySeconds: 10
        periodSeconds: 5
        failureThreshold: 30  # 10 + (5 × 30) = up to 160s to start
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 10
        failureThreshold: 3

Problem 5: HTTPS Health Check with Self-Signed Certificate

The Symptom

Health check fails when target group uses HTTPS protocol. Error: connection refused or SSL handshake failure.

Why It Happens

Your app serves HTTPS with a self-signed certificate, and the ALB can't verify it.

The Fix

ALB actually accepts self-signed certificates for health checks. The issue is usually that your app isn't listening on HTTPS at all, or it's on a different port.

Best practice: use HTTP for health checks between ALB and targets (ALB terminates SSL):

Client → HTTPS → ALB (SSL termination) → HTTP → Target
                                          ↑
                              Health check (HTTP)

aws elbv2 modify-target-group \
  --target-group-arn arn:aws:elasticloadbalancing:... \
  --health-check-protocol HTTP \
  --health-check-port traffic-port

Problem 6: EKS ALB Controller — Pods Not Registering

The Symptom

You deployed an Ingress with the ALB controller, but the target group has 0 registered targets.

Debug Steps

# Check ALB controller logs
kubectl logs -n kube-system deployment/aws-load-balancer-controller
 
# Check Ingress status
kubectl describe ingress my-ingress
 
# Common errors:
# - "failed to build LoadBalancer configuration" → annotation issue
# - "no backends" → service selector doesn't match pods
# - "target type ip requires VPC CNI" → node security group issue

Common Fixes

Ensure pods have readinessProbe:

The ALB controller only registers pods that pass their readiness probe:

spec:
  containers:
    - name: app
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 10

Check Ingress annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/healthcheck-port: "8080"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...

Quick Debug Checklist

# 1. Check target health
aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:...
 
# 2. Test health check from the target itself
curl -v http://localhost:8080/health
 
# 3. Check security groups allow ALB → Target traffic
aws ec2 describe-security-groups --group-ids sg-target
 
# 4. Verify target group configuration
aws elbv2 describe-target-groups --target-group-arns arn:aws:...
 
# 5. Check ALB access logs (enable if not already)
aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn arn:aws:elasticloadbalancing:... \
  --attributes Key=access_logs.s3.enabled,Value=true \
  Key=access_logs.s3.bucket,Value=my-alb-logs
 
# 6. For EKS: check ALB controller
kubectl logs -n kube-system deployment/aws-load-balancer-controller --tail=50

Prevention

1. Always define a /health endpoint in every application — simple, fast, no auth required
2. Use traffic-port for health check port unless you have a specific reason not to
3. Set reasonable thresholds — don't use defaults blindly
4. Test health checks during CI — curl the health endpoint in your integration tests
5. Monitor target health — CloudWatch alarm on UnHealthyHostCount > 0

For deeper AWS networking and EKS training, KodeKloud's AWS courses cover ALB configuration and EKS networking in hands-on lab environments. And if you need a cloud environment to practice, DigitalOcean is great for setting up load balancer experiments affordably.

502 from the ALB almost always means unhealthy targets. Fix the health check, fix the 502.