sentry-setup-ai-monitoring

What this skill does

# SOURCE: getsentry/sentry-for-claude
# PATH: skills/sentry-setup-ai-monitoring/SKILL.md
# DO NOT EDIT: This file is synced from external source


# Setup Sentry AI Agent Monitoring

This skill helps configure Sentry's AI Agent Monitoring to track LLM calls, agent executions, tool usage, and token consumption.

## When to Use This Skill

Invoke this skill when:
- User asks to "setup AI monitoring" or "add AI agent tracking"
- User wants to "monitor LLM calls" or "track OpenAI/Anthropic usage"
- User requests "AI observability" or "agent monitoring"
- User mentions tracking token usage, model latency, or AI costs
- User asks about instrumenting their AI/LLM code with Sentry

## CRITICAL: Detection-First Approach

**ALWAYS detect installed AI SDKs before suggesting configuration.** Do not assume which AI library the user is using.

---

## Step 1: Detect Platform and AI SDKs

### For JavaScript/TypeScript Projects

Run these commands to detect installed AI packages:

```bash
# Check package.json for AI SDKs
grep -E '"(openai|@anthropic-ai/sdk|ai|@langchain|@google/genai|@langchain/langgraph)"' package.json
```

**Supported JavaScript AI SDKs:**

| Package | Sentry Integration | Min SDK Version | Package Version |
|---------|-------------------|-----------------|-----------------|
| `openai` | `openAIIntegration` | 10.2.0 | >=4.0.0 <7 |
| `@anthropic-ai/sdk` | `anthropicAIIntegration` | 10.12.0 | >=0.19.2 <1 |
| `ai` (Vercel AI SDK) | `vercelAIIntegration` | 10.6.0 | >=3.0.0 <6 |
| `@langchain/*` | `langChainIntegration` | 10.22.0 | >=0.1.0 <1 |
| `@google/genai` | `googleGenAIIntegration` | 10.14.0 | >=0.10.0 <2 |
| `@langchain/langgraph` | `langGraphIntegration` | 10.25.0 | >=0.2.0 <1 |

### For Python Projects

```bash
# Check requirements.txt or pyproject.toml
grep -E '(openai|anthropic|langchain|huggingface)' requirements.txt pyproject.toml 2>/dev/null
```

**Supported Python AI SDKs:**

| Package | Sentry Extra | Min SDK Version | Package Version |
|---------|-------------|-----------------|-----------------|
| `openai` | `sentry-sdk[openai]` | 2.41.0 | >=1.0.0 |
| `anthropic` | `sentry-sdk[anthropic]` | 2.x | >=0.16.0 |
| `langchain` | `sentry-sdk[langchain]` | 2.x | >=0.1.11 |
| `huggingface_hub` | `sentry-sdk[huggingface_hub]` | 2.x | >=0.22.0 |

---

## Step 2: Verify Sentry SDK Version

### JavaScript
```bash
grep -E '"@sentry/(nextjs|react|node)"' package.json
```

Check version meets minimum for detected AI SDK (see table above).

**Upgrade if needed:**
```bash
npm install @sentry/nextjs@latest  # or appropriate package
```

### Python
```bash
pip show sentry-sdk | grep Version
```

**Upgrade if needed:**
```bash
pip install --upgrade "sentry-sdk[openai]"  # include detected extras
```

---

## Step 3: Verify Tracing is Enabled

AI Agent Monitoring requires tracing. Check that `tracesSampleRate` is set in `Sentry.init()`.

If not enabled, inform the user:
```
AI Agent Monitoring requires tracing to be enabled. I'll add tracesSampleRate to your Sentry configuration.
```

---

## Step 4: Configure Based on Detected SDK

### IMPORTANT: Present Detection Results First

Before configuring, tell the user what you found:

```
I detected the following AI SDK(s) in your project:
- [SDK NAME] (version X.X.X)

Sentry has automatic integration support for this SDK. I'll configure the appropriate integration.
```

If no supported SDK is detected:
```
I didn't detect any AI SDKs with automatic Sentry integration support in your project.

Your options:
1. Manual Instrumentation - I can help you set up custom spans for your AI calls
2. Install a supported SDK - If you're planning to add one of the supported SDKs

Would you like to proceed with manual instrumentation? If so, please describe where your AI/LLM calls are located.
```

---

## JavaScript Integration Configurations

### OpenAI Integration

**Automatic** - Enabled by default in SDK 10.2.0+

**Explicit configuration** (to enable input/output recording):

```javascript
import * as Sentry from "@sentry/nextjs"; // or @sentry/node, @sentry/react

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.openAIIntegration({
      recordInputs: true,  // Capture prompts/messages
      recordOutputs: true, // Capture responses
    }),
  ],
});
```

**Supported methods (auto-instrumented):**
- `chat.completions.create()`
- `responses.create()`

**Example usage (no changes needed):**
```javascript
import OpenAI from "openai";

const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello!" }],
});
// Automatically captured by Sentry
```

---

### Anthropic Integration

**Automatic** - Enabled by default in SDK 10.12.0+

**Explicit configuration:**

```javascript
import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.anthropicAIIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});
```

**Supported methods (auto-instrumented):**
- `messages.create()` / `messages.stream()`
- `messages.countTokens()`
- `completions.create()`
- `beta.messages.create()`

---

### Vercel AI SDK Integration

**Automatic in Node runtime** - Enabled by default in SDK 10.6.0+

**For Edge runtime (Next.js)** - Must be explicitly enabled in `sentry.edge.config.ts`:

```javascript
import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,
  integrations: [Sentry.vercelAIIntegration()],
});
```

**IMPORTANT: Telemetry must be enabled per-call:**

```javascript
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const result = await generateText({
  model: openai("gpt-4o"),
  prompt: "Hello!",
  experimental_telemetry: {
    isEnabled: true,
    recordInputs: true,
    recordOutputs: true,
    functionId: "my-generation", // Optional but recommended
  },
});
```

**Configuration options:**
```javascript
Sentry.vercelAIIntegration({
  recordInputs: true,
  recordOutputs: true,
  force: false, // Force enable even if AI module not detected
})
```

---

### LangChain Integration

**Requires explicit integration** in SDK 10.22.0+

```javascript
import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.langChainIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});
```

**Auto-instrumented operations:**
- Chat model calls (`gen_ai.request`)
- LLM pipeline executions (`gen_ai.pipeline`)
- Chain invocations (`gen_ai.invoke_agent`)
- Tool calls (`gen_ai.execute_tool`)
- `invoke()`, `stream()`, `batch()` methods

---

### LangGraph Integration

**Requires explicit integration** in SDK 10.25.0+

```javascript
import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.langGraphIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});
```

**Auto-instrumented:**
- Agent creation (StateGraph compilation)
- Agent invocation (`invoke()` method)

---

### Google GenAI Integration

**Automatic** - Enabled by default in SDK 10.14.0+

**Explicit configuration:**

```javascript
import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.googleGenAIIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});
```

**Supported methods:**
- `models.generateContent()` / `models.generateContentStream()`
- `chats.create()`
- `sendMessage()` / `sendMessageStream()`

---

## Python Integration Configurations

### OpenAI Integration (Python)

**Install with extra:**
```bash
pip install "sentry-sdk[openai]"
```

**Configuration:**

```python
import sentry_sdk
from sentry_sdk.integrations.openai import OpenAIIntegration

sentry_sdk.init(
    dsn="YOUR_DSN_HERE",