runbookhermes-aiops-agent

What this skill does

```markdown
---
name: runbookhermes-aiops-agent
description: Hermes-native AIOps agent for evidence-driven incident response, approval-gated remediation, and runbook learning on payment and production systems.
triggers:
  - set up RunbookHermes for incident response
  - add approval-gated remediation to my AIOps agent
  - integrate Prometheus and Loki with runbook agent
  - create a runbook skill from an incident
  - configure evidence-driven root cause analysis
  - set up Alertmanager webhook for incident intake
  - add human-in-the-loop approval for production actions
  - build a Hermes AIOps agent with runbook learning
---

# RunbookHermes AIOps Agent

> Skill by [ara.so](https://ara.so) — Daily 2026 Skills collection.

RunbookHermes is a Hermes-native AIOps agent for payment and production incident response. It extends the Hermes Agent runtime with evidence collection (Prometheus, Loki, Jaeger), an `EvidenceStack` context engine, `IncidentMemory`, approval-gated remediation, checkpoint/rollback safety gates, and runbook skill generation — turning one-off incident handling into accumulated operational knowledge.

---

## Installation

### Prerequisites

- Python 3.11+
- Node.js 18+ (for the Web Console static assets)
- Docker (optional, for the reference payment demo environment)

### Clone and Install

```bash
git clone https://github.com/Tommy-yw/RunbookHermes.git
cd RunbookHermes
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

### Install the Web Console dependencies (optional)

```bash
cd web
npm install
npm run build
cd ..
```

### Start the API + Web Console

```bash
uvicorn apps.runbook_api.main:app --host 0.0.0.0 --port 8080 --reload
```

Navigate to `http://localhost:8080` for the AIOps Web Console.

---

## Configuration

All production integrations are controlled by environment variables. Copy `.env.example` to `.env` and fill in your values.

```bash
cp .env.example .env
```

### Core environment variables

```dotenv
# ── Model provider (OpenAI-compatible endpoint) ──────────────────────────────
OPENAI_API_KEY=$OPENAI_API_KEY
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=gpt-4o

# ── Observability backends ────────────────────────────────────────────────────
PROMETHEUS_URL=http://prometheus:9090
LOKI_URL=http://loki:3100
JAEGER_URL=http://jaeger:16686

# ── Deployment / execution backend ───────────────────────────────────────────
DEPLOY_BACKEND=custom_http          # or: kubernetes, argocd
DEPLOY_API_URL=http://deploy-api/v1
DEPLOY_API_TOKEN=$DEPLOY_API_TOKEN

# ── Notification integrations ────────────────────────────────────────────────
FEISHU_APP_ID=$FEISHU_APP_ID
FEISHU_APP_SECRET=$FEISHU_APP_SECRET
WECOM_CORP_ID=$WECOM_CORP_ID
WECOM_AGENT_ID=$WECOM_AGENT_ID
WECOM_SECRET=$WECOM_SECRET

# ── Approval gate ────────────────────────────────────────────────────────────
APPROVAL_REQUIRED=true              # set false for dry-run-only mode
APPROVAL_TIMEOUT_SECONDS=300

# ── Memory / persistence ─────────────────────────────────────────────────────
INCIDENT_MEMORY_BACKEND=sqlite      # or: postgres, redis
INCIDENT_MEMORY_DSN=sqlite:///data/incident_memory.db

# ── Mock / demo fallback ─────────────────────────────────────────────────────
USE_MOCK_OBSERVABILITY=true         # set false when real backends are reachable
```

Load the `.env` before running:

```bash
export $(cat .env | xargs)
uvicorn apps.runbook_api.main:app --reload
```

---

## Key Concepts

| Concept | Description |
|---|---|
| **EvidenceStack** | Structured context engine — organises alert, evidence, hypotheses, action plan, and final answer instead of dumping raw logs into the prompt. |
| **IncidentMemory** | Domain-specific memory provider — stores service profiles, team preferences, incident summaries, recurring root causes, and runbook skills. |
| **Approval Gate** | All write/destructive actions pass through: policy check → approval request → checkpoint → dry-run → controlled execution → recovery verification. |
| **Runbook Skill** | A reusable, machine-readable operational procedure generated from a successfully resolved incident. |
| **Profile** | `profiles/runbook-hermes/` — the Hermes agent profile that wires together tools, memory, context engine, and persona for AIOps. |

---

## Creating and Managing Incidents

### Via REST API

```python
import httpx

BASE = "http://localhost:8080"

# Create an incident manually
resp = httpx.post(f"{BASE}/api/incidents", json={
    "service": "payment-service",
    "severity": "critical",
    "title": "HTTP 503 spike on /v1/charge",
    "description": "Error rate exceeded 40% for 5 minutes",
    "source": "api",
})
incident = resp.json()
incident_id = incident["id"]
print(f"Created incident: {incident_id}")

# Fetch incident detail (evidence + root cause)
detail = httpx.get(f"{BASE}/api/incidents/{incident_id}").json()
print(detail["executive_summary"])
```

### Via Alertmanager Webhook

Configure Alertmanager to POST to the RunbookHermes gateway:

```yaml
# alertmanager.yml
receivers:
  - name: runbookhermes
    webhook_configs:
      - url: http://runbook-hermes:8080/gateway/alertmanager
        send_resolved: true
```

### Via Feishu / WeCom Event Callback

```python
# The gateway shells are registered automatically.
# Point your Feishu app "Event Subscription → Request URL" to:
#   http://runbook-hermes:8080/gateway/feishu/event
# Point WeCom "Receive Messages URL" to:
#   http://runbook-hermes:8080/gateway/wecom/event
```

---

## Evidence Collection

### Using Observability Adapters Directly

```python
from integrations.observability.prometheus import PrometheusAdapter
from integrations.observability.loki import LokiAdapter
from integrations.observability.trace import TraceAdapter

prom = PrometheusAdapter(base_url="http://prometheus:9090")
loki = LokiAdapter(base_url="http://loki:3100")
trace = TraceAdapter(base_url="http://jaeger:16686")

# Query error rate for the last 10 minutes
metrics = await prom.query_range(
    query='rate(http_requests_total{service="payment-service",status=~"5.."}[1m])',
    start="-10m",
    end="now",
    step="30s",
)

# Fetch recent error logs
logs = await loki.query_logs(
    query='{service="payment-service"} |= "ERROR"',
    limit=100,
    since="10m",
)

# Fetch a trace by ID
trace_data = await trace.get_trace(trace_id="abc123def456")
```

### Using the EvidenceStack Context Engine

```python
from plugins.context_engine.evidence_stack.stack import EvidenceStack

stack = EvidenceStack(incident_id="INC-20260501-001")

# Add alert trigger
stack.add_alert(
    title="HTTP 503 spike",
    service="payment-service",
    severity="critical",
    labels={"env": "prod", "region": "us-east-1"},
)

# Add collected evidence
stack.add_evidence(
    evidence_id="EV-001",
    kind="metric",
    summary="Error rate: 42% over last 5 min",
    source="prometheus",
    raw_ref="prometheus:query:rate(http_requests_total...)",  # not inlined
)

stack.add_evidence(
    evidence_id="EV-002",
    kind="log",
    summary="500+ 'upstream connect error' log lines in payment-service",
    source="loki",
)

# Add a hypothesis
stack.add_hypothesis(
    hypothesis="Upstream Redis connection pool exhausted, causing cascade 503s",
    confidence=0.85,
    supporting_evidence=["EV-001", "EV-002"],
)

# Render compressed context for model prompt
context = stack.render_for_prompt(max_tokens=2048)
print(context)
```

---

## Approval-Gated Remediation

### Submitting a Risky Action for Approval

```python
from runbook_hermes.approval import ApprovalGate, ActionRisk

gate = ApprovalGate()

# Register a high-risk action
approval_request = await gate.request_approval(
    incident_id="INC-20260501-001",
    action="rollback",
    risk=ActionRisk.HIGH,
    payload={
        "service": "payment-service",
        "target_version": "v2.3.1",
        "current_version": "v2.4.0",
    },
    dry_run_result={
        "would_affect": ["payment-service:prod:us-east-1"],
        "estimated_downt