AWS RDS max_connections exceeded under load

Production went down. RDS PostgreSQL was throwing:

FATAL: remaining connection slots are reserved for non-replication superuser connections

We had 50 application pods, each with a connection pool of 10 connections = 500 connections. RDS db.t3.medium max_connections is ~170.

Immediate fix (10 minutes):

Reduced POOL_SIZE env var in the app deployment from 10 to 3 temporarily:

bash

kubectl set env deployment/api POOL_SIZE=3

Rollout happened in 2 minutes. Connections dropped to ~150. RDS recovered.

Proper fix (25 minutes):

Set up PgBouncer as a connection pooler in front of RDS:

yaml

# PgBouncer Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgbouncer
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: pgbouncer
          image: pgbouncer/pgbouncer:latest
          env:
            - name: DATABASES_HOST
              value: "my-rds.cluster-xxxx.us-east-1.rds.amazonaws.com"
            - name: PGBOUNCER_POOL_MODE
              value: "transaction"
            - name: PGBOUNCER_MAX_CLIENT_CONN
              value: "1000"
            - name: PGBOUNCER_DEFAULT_POOL_SIZE
              value: "20"

Now all 50 pods connect to PgBouncer (max 1000 client connections). PgBouncer maintains just 20 connections to RDS. Problem solved.

Lesson: RDS max_connections is calculated from instance RAM. Never let your total app connection pool exceed it. Use PgBouncer for any app with more than a handful of replicas.