At 3:17 AM on October 14th, 2025, our client's database connection pool hit its limit. In 2024, this would have triggered a PagerDuty alert, woken up an on-call engineer, initiated a twenty-minute diagnosis, and required a manual parameter change and service restart. In 2025, the autonomous operations center detected the anomaly, diagnosed the root cause from Grafana metrics, increased the connection pool limit via the AWS Parameter Store API, restarted the affected service via kubectl, and posted a full incident report to Slack — all in 4 minutes and 12 seconds. Nobody was woken up.
This is AIOps in production. Not a demo. Not a proof of concept. A real system, handling real incidents, in a real production environment. Here is exactly how we built it.
Architecture Overview
The Autonomous Ops Center (AOC) has four components: a Signal Layer (Prometheus + Grafana + PagerDuty), an Intelligence Layer (n8n workflows with OpenAI function calling), an Action Layer (a sandboxed execution environment for remediation scripts), and a Documentation Layer (automatic runbook generation and Slack reporting). These communicate through a central event bus (we use AWS EventBridge).
Why n8n Instead of Custom Code?
n8n gives you visual workflow orchestration, a library of 400+ integrations, and built-in error handling and retry logic. Building this same orchestration layer in custom Python takes 4–6 weeks. n8n delivers it in a weekend, and every workflow is auditable by engineers who are not Python developers. For an operations platform where trust and transparency matter, that's a significant advantage.
The Five Core Workflows
Workflow 1: Intelligent Incident Triage
When PagerDuty fires, the triage workflow automatically: fetches the last 30 minutes of metrics from Prometheus, pulls recent deployment history from ArgoCD, queries the last 7 days of similar incidents from our incident database, and sends all of this context to GPT-4o with a prompt asking for: likely root cause, remediation options ranked by risk, and whether human escalation is recommended.
// n8n Function node — AI Triage Prompt Construction
{
"model": "gpt-4o",
"messages": [
{
"role": "system",
"content": "You are a senior SRE. Analyze the following incident context and provide:\n1. Most likely root cause (with confidence %)\n2. Top 3 remediation options ordered by risk (low/medium/high)\n3. ASSESS: Can this be auto-remediated safely? (yes/no/maybe)\n4. Estimated blast radius if unresolved\n\nIMPORTANT: Never recommend actions that would affect more than 10% of traffic. If uncertain, recommend human escalation."
},
{
"role": "user",
"content": "INCIDENT: {{ $json.incident.title }}\n\nMETRICS (last 30min):\n{{ $json.metrics_summary }}\n\nRECENT DEPLOYMENTS:\n{{ $json.recent_deployments }}\n\nSIMILAR PAST INCIDENTS:\n{{ $json.similar_incidents }}"
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "triage_result",
"schema": {
"root_cause": { "type": "string" },
"confidence": { "type": "number" },
"remediation_options": { "type": "array" },
"auto_remediable": { "type": "boolean" },
"blast_radius": { "type": "string" }
}
}
}
}Workflow 2: Autonomous Remediation
For incidents the AI classifies as auto-remediable with high confidence, the remediation workflow executes predefined runbook scripts in a sandboxed environment. Every remediation action is: logged to an immutable audit trail, executed in a dry-run mode first with the output verified by the AI, rate-limited (maximum 3 auto-remediations per hour to prevent cascade failures), and reversible (every script has a corresponding rollback script).
Workflow 3: Capacity Forecasting
Every 6 hours, the forecasting workflow analyzes resource utilization trends across all workloads, identifies resources that will breach thresholds in the next 24/48/72 hours, and generates pre-emptive scaling recommendations. Human approval is required for capacity changes that affect production. The workflow drafts the change, the engineer approves with one click.
Workflow 4: Cost Anomaly Response
AWS Cost Anomaly Detection fires a webhook when spend exceeds normal patterns. The workflow automatically correlates cost anomalies with recent deployments, infrastructure changes, and traffic patterns, then generates a PDF cost forensics report with specific remediation recommendations. The median time to identify a cost anomaly root cause dropped from 3 days to 22 minutes.
Workflow 5: Runbook Generation
Every incident automatically generates a draft runbook entry: what happened, why it happened, how it was resolved, and how to prevent recurrence. These drafts go to a Notion database for human review and enrichment. After 8 months, we have 340 AI-generated runbook entries covering virtually every class of incident we've seen. The on-call rotation has become dramatically calmer.
Production Result
After 8 months in production: 71% of P3/P4 incidents auto-resolved without human intervention. Mean time to resolution for P1/P2 incidents decreased from 47 minutes to 12 minutes (AI-assisted diagnosis). On-call engineer wakeups decreased by 68%. Zero auto-remediation incidents caused additional outages.
Critical Safety Note
Never give an autonomous system the ability to scale down production infrastructure, delete resources, or modify network security groups without human approval. Auto-remediation scope should be strictly limited to operations that are low-risk and fully reversible: service restarts, parameter adjustments, cache flushes, scaling up (never down).
The autonomous operations center is not about replacing your SRE team. It's about making them dramatically more effective — handling the noise so they can focus on the signal, the complex incidents, the architectural improvements that make systems more reliable in the first place. Your best engineers should be doing their most valuable work, not restarting services at 3 AM.