πŸŽ‰ DevOps Interview Prep Bundle is live β€” 1000+ Q&A across 20 topicsGet it β†’
All Articles

Agentic DevOps: How AI Agents Will Autonomously Manage Infrastructure in 2026

AI agents that detect incidents, diagnose root causes, execute remediation, and write postmortems without human intervention are already running in production. Here is what agentic DevOps looks like and where it is heading.

Shubham6 min read
Share:Tweet

The shift from AI-assisted DevOps to agentic DevOps is already happening. The difference is significant: AI-assisted means a human asks Claude "why is this pod failing?" and gets an answer. Agentic means the system detects the failure, diagnoses it, fixes it, and files the incident report β€” all without waking anyone up.

This is not science fiction. Teams at major tech companies are running agentic infrastructure management in production today. Here is what it looks like, how it works, and what it means for DevOps engineers.

What Makes Something "Agentic"

An AI agent has three properties that distinguish it from a simple LLM call:

  1. Tool use β€” it can take actions, not just generate text
  2. Memory β€” it remembers what it did and what it learned
  3. Autonomy β€” it decides what to do next without being told

A ChatGPT conversation is not agentic. A system that receives a PagerDuty alert, runs kubectl describe, checks Prometheus metrics, decides the fix, applies it, and verifies recovery β€” that is agentic.

The Agentic DevOps Stack

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   Trigger Layer                  β”‚
β”‚  PagerDuty Β· Alertmanager Β· Scheduled Β· Webhook  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                      β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   Agent Core                     β”‚
β”‚         Claude API + Tool Use Loop               β”‚
β”‚   Observe β†’ Think β†’ Act β†’ Verify β†’ Remember      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
           β”‚
    β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚              Tool Belt              β”‚
    β”‚  kubectl  β”‚  Prometheus API         β”‚
    β”‚  AWS SDK  β”‚  GitHub API             β”‚
    β”‚  Terraformβ”‚  Slack/PagerDuty        β”‚
    β”‚  Runbooks β”‚  Incident DB            β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Real Agentic Loop: Incident Remediation

Here is an actual agentic loop running in production at scale:

python
import anthropic
import json
from datetime import datetime
 
client = anthropic.Anthropic()
 
TOOLS = [
    {
        "name": "kubectl_get",
        "description": "Run kubectl get on a resource",
        "input_schema": {
            "type": "object",
            "properties": {
                "resource": {"type": "string"},
                "namespace": {"type": "string"},
                "flags": {"type": "string"}
            },
            "required": ["resource"]
        }
    },
    {
        "name": "kubectl_describe",
        "description": "Run kubectl describe on a specific resource",
        "input_schema": {
            "type": "object",
            "properties": {
                "resource": {"type": "string"},
                "name": {"type": "string"},
                "namespace": {"type": "string"}
            },
            "required": ["resource", "name"]
        }
    },
    {
        "name": "kubectl_rollout_restart",
        "description": "Restart a deployment",
        "input_schema": {
            "type": "object",
            "properties": {
                "deployment": {"type": "string"},
                "namespace": {"type": "string"}
            },
            "required": ["deployment", "namespace"]
        }
    },
    {
        "name": "query_prometheus",
        "description": "Run a PromQL query",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string"},
                "time_range": {"type": "string"}
            },
            "required": ["query"]
        }
    },
    {
        "name": "resolve_pagerduty_incident",
        "description": "Resolve a PagerDuty incident with a note",
        "input_schema": {
            "type": "object",
            "properties": {
                "incident_id": {"type": "string"},
                "resolution_note": {"type": "string"}
            },
            "required": ["incident_id", "resolution_note"]
        }
    },
    {
        "name": "create_postmortem",
        "description": "Create a postmortem document in Confluence/Notion",
        "input_schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "timeline": {"type": "string"},
                "root_cause": {"type": "string"},
                "remediation": {"type": "string"},
                "action_items": {"type": "string"}
            },
            "required": ["title", "root_cause", "remediation"]
        }
    }
]
 
 
def execute_tool(name: str, inputs: dict) -> str:
    """Execute a tool and return the result as a string."""
    import subprocess
 
    if name == "kubectl_get":
        cmd = ["kubectl", "get", inputs["resource"]]
        if "namespace" in inputs:
            cmd += ["-n", inputs["namespace"]]
        if "flags" in inputs:
            cmd += inputs["flags"].split()
        cmd += ["-o", "wide"]
        result = subprocess.run(cmd, capture_output=True, text=True)
        return result.stdout or result.stderr
 
    elif name == "kubectl_describe":
        cmd = ["kubectl", "describe", inputs["resource"], inputs["name"]]
        if "namespace" in inputs:
            cmd += ["-n", inputs["namespace"]]
        result = subprocess.run(cmd, capture_output=True, text=True)
        return result.stdout[-3000:] if len(result.stdout) > 3000 else result.stdout
 
    elif name == "kubectl_rollout_restart":
        cmd = ["kubectl", "rollout", "restart",
               f"deployment/{inputs['deployment']}",
               "-n", inputs["namespace"]]
        result = subprocess.run(cmd, capture_output=True, text=True)
        return result.stdout or result.stderr
 
    elif name == "query_prometheus":
        import requests
        PROM_URL = "http://prometheus.monitoring.svc:9090"
        resp = requests.get(f"{PROM_URL}/api/v1/query",
                           params={"query": inputs["query"]})
        data = resp.json()
        return json.dumps(data.get("data", {}).get("result", [])[:5], indent=2)
 
    elif name == "resolve_pagerduty_incident":
        # PagerDuty API call would go here
        return f"Incident {inputs['incident_id']} resolved: {inputs['resolution_note']}"
 
    elif name == "create_postmortem":
        # Confluence/Notion API call would go here
        return f"Postmortem created: {inputs['title']}"
 
    return f"Tool {name} not implemented"
 
 
def run_agentic_incident_response(incident: dict) -> dict:
    """
    Full agentic loop: receive incident β†’ diagnose β†’ remediate β†’ document.
    Returns summary of what was done.
    """
    system_prompt = """You are an autonomous SRE agent. When given an incident,
you must:
1. Investigate using available tools
2. Identify the root cause
3. Apply the fix (use kubectl tools)
4. Verify the fix worked
5. Resolve the incident and create a postmortem
 
Be methodical. Explain your reasoning before each tool call.
Only apply fixes you are confident about.
If unsure, gather more data first."""
 
    messages = [
        {
            "role": "user",
            "content": f"Incident received:\n{json.dumps(incident, indent=2)}\n\nInvestigate and resolve this incident."
        }
    ]
 
    action_log = []
    max_iterations = 15
 
    for iteration in range(max_iterations):
        response = client.messages.create(
            model="claude-sonnet-5",
            max_tokens=2000,
            system=system_prompt,
            tools=TOOLS,
            messages=messages
        )
 
        # Add assistant response to conversation
        messages.append({"role": "assistant", "content": response.content})
 
        # Check if done
        if response.stop_reason == "end_turn":
            break
 
        # Process tool calls
        tool_results = []
        for block in response.content:
            if block.type == "tool_use":
                print(f"  Agent running: {block.name}({block.input})")
                result = execute_tool(block.name, block.input)
                action_log.append({
                    "tool": block.name,
                    "input": block.input,
                    "result": result[:500],
                    "timestamp": datetime.utcnow().isoformat()
                })
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": result
                })
 
        if tool_results:
            messages.append({"role": "user", "content": tool_results})
        else:
            break
 
    # Extract final summary from last text response
    final_text = ""
    for block in response.content:
        if hasattr(block, "text"):
            final_text = block.text
 
    return {
        "incident_id": incident.get("id"),
        "actions_taken": action_log,
        "final_summary": final_text,
        "iterations": iteration + 1
    }

What Agentic Systems Can Do Today

Already in production at scale:

  • Auto-restart pods on OOMKilled or CrashLoopBackOff with root cause analysis
  • Scale deployments when HPA metrics show sustained high load
  • Rotate secrets when expiry alerts fire
  • Revert bad deploys when error rates spike post-deployment
  • Generate and file postmortems automatically after incidents resolve

Emerging in 2026:

  • Proactive remediation β€” fix issues before alerts fire, based on trend detection
  • Cross-system diagnosis β€” correlate K8s events + APM traces + logs + deploys automatically
  • Autonomous cost optimization β€” rightsizing, spot instance management, idle resource cleanup

What Changes for DevOps Engineers

The common fear is job replacement. The reality is role evolution.

What gets automated:

  • Tier-1 incident response (restart pods, clear queues, scale replicas)
  • Routine runbook execution
  • Infrastructure health checks
  • Postmortem first drafts

What becomes more important:

  • Designing the agent systems and their guardrails
  • Writing and maintaining runbooks the agents use
  • Defining what agents are and are not allowed to do autonomously
  • Reviewing agent decisions and improving them over time
  • Handling novel incidents agents have never seen

The DevOps engineer of 2027 is less "execute the runbook" and more "design the system that executes runbooks and know when to override it."

Safety Guardrails You Must Build

Autonomous systems need hard limits:

python
AUTONOMOUS_ALLOWED = [
    "kubectl rollout restart",
    "kubectl scale (replicas increase only)",
    "pod deletion (not deployment deletion)",
    "alert silencing (< 2 hours)",
]
 
REQUIRES_HUMAN_APPROVAL = [
    "kubectl delete deployment",
    "terraform apply",
    "database operations",
    "scale down (replicas decrease)",
    "any production secret rotation",
]
 
NEVER_AUTONOMOUS = [
    "kubectl delete namespace",
    "terraform destroy",
    "IAM policy changes",
    "network policy changes",
]

The teams winning with agentic DevOps are not the ones who gave agents the most power β€” they are the ones who drew the clearest lines around what agents can and cannot touch.


Building agentic systems? Read our AI SRE incident commander with Claude API and self-healing Kubernetes with AI agents.

πŸ”§

Today I Fixed

Short real fixes from production β€” posted daily

Browse fixes
Newsletter

Stay ahead of the curve

Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam β€” just practical engineering content.

Related Articles

Comments