🎉 DevOps Interview Prep Bundle is live — 1000+ Q&A across 20 topicsGet it →
All Articles

Autonomous Chaos Engineering: AI Agents That Design Their Own Experiments in 2026

Manually writing chaos experiments requires knowing your system's weak points in advance — which defeats some of the purpose. AI agents that analyze architecture and telemetry to autonomously design targeted, safe chaos experiments are emerging as the next step in chaos engineering practice.

Shubham4 min read
Share:Tweet

Chaos engineering has always had a chicken-and-egg problem: the experiments most worth running are the ones targeting weaknesses you don't already know about, but designing a good experiment requires understanding the system well enough to predict where it might break. An agent that reads your architecture, dependency graph, and historical incidents can surface experiment candidates a human wouldn't think to write, because it isn't limited to the mental model the on-call engineer happens to carry around.

From Manual Experiment Design to Agent-Suggested

Traditional chaos engineering:
  Engineer thinks about what might break → writes a specific experiment
  (kill this pod, add latency to this service) → runs it → learns

Agent-assisted:
  Agent analyzes service dependency graph + historical incidents +
  current architecture → identifies untested failure modes →
  proposes ranked experiment candidates → HUMAN reviews and approves
  → experiment runs (via existing chaos tooling — Litmus, Chaos Mesh, Gremlin)

The agent's contribution is breadth of hypothesis generation across a system too large for one engineer to hold fully in their head — not running the actual experiment, which stays on established chaos engineering tooling.

Dependency and Risk Analysis

python
import anthropic
import json
 
client = anthropic.Anthropic()
 
 
def gather_system_context(service_name: str) -> dict:
    return {
        "dependency_graph": get_service_dependencies(service_name),    # from service mesh / tracing data
        "past_incidents": get_incident_history(service_name, months=12),
        "current_redundancy": get_replica_and_az_distribution(service_name),
        "existing_chaos_tests_run": get_chaos_experiment_history(service_name),
        "sla_criticality": get_service_criticality_tier(service_name),
    }

Experiment Candidate Generation

python
DESIGN_PROMPT = """Design chaos engineering experiment candidates for this service.
 
Service: {service_name}
Dependency graph (what it calls, what calls it): {dependency_graph}
Past incidents (last 12 months): {past_incidents}
Current redundancy (replicas, AZ distribution): {current_redundancy}
Chaos experiments already run on this service: {existing_tests}
Criticality tier: {sla_criticality}
 
Propose 3-5 experiment candidates that:
1. Target failure modes NOT already covered by existing_tests
2. Are informed by past_incidents — if a similar failure mode caused a
   real incident before, prioritize verifying it's actually fixed
3. Are appropriately scoped to criticality tier (don't propose a
   full-AZ failure test on a tier-1 payment service without extensive
   safeguards; simple pod-kill tests are fine at any tier)
 
For each candidate, specify:
- Hypothesis (what you expect to happen if the system is healthy)
- Blast radius and safety guardrails (max % of traffic affected, auto-abort conditions)
- Success criteria (what "the system handled this correctly" looks like)
 
Respond with ONLY valid JSON:
{{"candidates": [
  {{"name": "...", "hypothesis": "...", "blast_radius": "...",
    "safety_guardrails": "...", "success_criteria": "...", "priority": "high"|"medium"|"low"}}
]}}"""
 
 
def design_experiments(service_name: str, context: dict) -> dict:
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=2500,
        messages=[{
            "role": "user",
            "content": DESIGN_PROMPT.format(service_name=service_name, **context)
        }]
    )
    text = response.content[0].text.strip()
    if text.startswith("```"):
        text = text.split("```")[1].replace("json", "", 1).strip()
    return json.loads(text)

Example Output

Experiment Candidates: payments-api (Tier 1)

1. [HIGH] Database connection pool exhaustion simulation
   Hypothesis: connection pool exhaustion on the primary DB causes graceful
   degradation (queued requests, circuit breaker opens) rather than cascading
   timeouts to upstream callers.
   Rationale: past_incidents shows a connection pool exhaustion incident
   6 months ago — no chaos test has verified the fix actually holds.
   Blast radius: 5% of traffic, auto-abort if error rate exceeds 2%
   Success criteria: circuit breaker opens within 10s, upstream services
   receive fast-fail responses (not hung connections), no cascading timeout

2. [MEDIUM] Single-AZ dependency failure (Redis cache)
   Hypothesis: losing the Redis replica in one AZ causes a brief latency
   spike (cache miss fallback to DB) but no errors.
   Rationale: dependency_graph shows Redis is single-AZ, no test has verified
   the cache-miss fallback path works under real failure, not just code review.
   Blast radius: kill Redis pods in one AZ only, auto-abort if p99 exceeds 2s
   Success criteria: error rate stays at 0%, p99 latency recovers within 30s

Turning Candidates Into Actual Runnable Experiments

yaml
# Agent-proposed candidate translated into a Litmus ChaosEngine
# — a human reviews and approves this before it runs
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payments-db-pool-exhaustion
spec:
  appinfo:
    appns: production
    applabel: app=payments-api
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-network-latency
      spec:
        components:
          env:
            - name: TARGET_CONTAINER
              value: payments-api
            - name: NETWORK_LATENCY
              value: "2000"    # simulates slow DB responses -> pool exhaustion pressure
            - name: TOTAL_CHAOS_DURATION
              value: "60"

Why This Stays Human-Approved, Not Autonomous Execution

Proposing an experiment and running it against production are very different risk profiles — the agent's proposal step is low-risk (it's just generating hypotheses), but actually injecting failure into a live system always needs a human confirming the blast radius, timing (not during peak traffic or another ongoing incident), and abort conditions are correct for the current state of the system. The value of agent-assisted chaos engineering is breadth of hypothesis generation across a system too large for one person to fully model, not removing the human judgment call about when and how to actually break something on purpose.


More AI observability and reliability content? Read our Chaos engineering will become standard and Self-healing CI/CD pipelines: autonomous agents.

🔧

Today I Fixed

Short real fixes from production — posted daily

Browse fixes
Newsletter

Stay ahead of the curve

Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam — just practical engineering content.

Related Articles

Comments