deepeval-tracing

Included with Lifetime

$97 forever

Instrument an AI application with DeepEval's native tracing so its behavior is visible in Confident AI. TRIGGER when the user wants to add DeepEval tracing or @observe to an LLM app, agent, RAG pipeline, or chatbot; wire a framework, model-provider, or vector-database integration (LangGraph, LangChain, OpenAI Agents, LlamaIndex, Pydantic AI, CrewAI, and others); choose between a native integration and manual instrumentation; set span types, tags, or metadata; or send DeepEval-SDK traces to Confident AI's Observatory. DO NOT TRIGGER for building DeepEval pytest eval suites, datasets, goldens, metrics, or deepeval test run (use the `deepeval` skill), or for raw OpenTelemetry / OTLP export without the deepeval package (use the `deepeval-otel` skill). This skill is purely DeepEval-SDK instrumentation — producing well-formed traces, not running evals.

Backend & APIs

What this skill does

# DeepEval Tracing

Use this skill to instrument an **AI application** — an LLM app, agent, RAG
pipeline, or chatbot — with **DeepEval's native tracing** so its execution is
visible span by span in **Confident AI's Observatory**. The work is: pick a
supported integration when one exists, fall back to manual `@observe`
otherwise, give each span a meaningful type, and add tags and metadata.

This skill stops at producing well-formed traces. Attaching evaluation metrics
and running evals is the `deepeval` skill's job.

## Scope: AI Applications Only

Instrument only the AI parts of the system — agent loops and planning, LLM
calls, retrieval / vector search, and tool calls. The span types (`llm`,
`retriever`, `tool`, `agent`) describe AI components. Do not trace non-AI
software (web servers, CRUD backends, infrastructure). If the target has no
LLM, agent, retrieval, or tool-calling component, this skill does not apply.

## When to Use vs the `deepeval` and `deepeval-otel` Skills

- **This skill (`deepeval-tracing`)** — instrument an app with the DeepEval SDK
(`@observe`, framework integrations) so traces reach Confident AI.
- **`deepeval` skill** — build pytest eval suites: datasets, metrics, traced
evals, `deepeval test run`, iteration. It runs evals *against* an app this
skill instrumented.
- **`deepeval-otel` skill** — instrument with the vendor-neutral OpenTelemetry
SDK instead of the DeepEval SDK (raw OTLP, including non-Python apps).

The three are complementary. If unsure between this skill and `deepeval-otel`:
use this one when the app is Python and you want the DeepEval SDK; use
`deepeval-otel` when you want raw OpenTelemetry or the app is not Python.

## Prerequisites

- An AI application in Python with `pip install deepeval`.
- For traces to reach Confident AI: `deepeval login`, or an exported
`CONFIDENT_API_KEY` (preferred for CI and non-interactive runs).

## Workflow

1. Confirm the target is an AI application (it has LLM calls, an agent loop,
retrieval, or tool calls). If it has none of these, stop — this skill does
not apply.
2. Detect the framework, model provider, agent SDK, and vector database in use.
3. Read `references/integrations.md` and the exact integration doc for what was
detected. Prefer a native integration over manual instrumentation.
4. If no native integration fits, instrument manually with `@observe`. Read
`references/tracing.md`.
5. Give each span a meaningful `type` (`llm`, `retriever`, `tool`, `agent`) and
capture inputs/outputs.
6. Add trace-level tags and metadata where they help diagnose failure patterns.
Never trace secrets, credentials, or raw sensitive data.
7. Confirm `deepeval login` or `CONFIDENT_API_KEY`, then verify traces appear
in the Confident AI Observatory.

## Core Principles

1. Instrument AI components only — `llm`, `retriever`, `tool`, `agent` spans.
Never trace non-AI software.
2. Prefer a supported integration over manual `@observe`. Manual tracing is the
fallback for unsupported frameworks and app-owned wrapper boundaries.
3. Read the exact integration doc before writing tracing code.
4. Give spans meaningful types; let names default to function names unless
there is a strong reason to override.
5. Never trace secrets, credentials, API keys, or raw sensitive user data.
6. Producing traces is the scope. Attaching metrics and running evals belong to
the `deepeval` skill; raw OpenTelemetry export belongs to `deepeval-otel`.

## References

| Topic | File |
| --- | --- |
| Manual instrumentation: `@observe`, span types, tags, metadata | `references/tracing.md` |
| Integration selection rule and framework / model / vector-DB doc index | `references/integrations.md` |

Files: 4

Size: 14.8 KB

Complexity: 42/100

Category: Backend & APIs

Source: https://github.com/confident-ai/deepeval/tree/main/skills/deepeval-tracing

Related in Backend & APIs

jfrog

Included

Interact with the JFrog Platform via the JFrog CLI and REST/GraphQL APIs. Use this skill when the user wants to manage Artifactory repositories, upload or download artifacts, manage builds, configure permissions, manage users and groups, work with access tokens, configure JFrog CLI servers, search artifacts, manage properties, set up replication, manage JFrog Projects, run security audits or scans, look up CVE details, query exposures scan results from JFrog Advanced Security, manage release bundles and lifecycle operations, aggregate or export platform data, or perform any JFrog Platform administration task. Also use when the user mentions jf, jfrog, artifactory, xray, distribution, evidence, apptrust, onemodel, graphql, workers, mission control, curation, advanced security, exposures, or any JFrog product name.

Backend & APIsscripts

cupynumeric-migration-readiness

Included

Pre-migration readiness assessor for porting NumPy to cuPyNumeric. Use BEFORE substantial porting work begins when the user asks whether code will scale on GPU, whether they should migrate to cuPyNumeric, which NumPy patterns transfer cleanly, what must be refactored before porting, or mentions pre-port assessment, scaling analysis, or refactor planning. Inspect the user's source code, look up NumPy usage, cross-reference the cuPyNumeric API support manifest, and distinguish distributed-scaling-friendly patterns from blockers such as unsupported APIs, scalar synchronization, host round-trips, Python/object-heavy control flow, shape/data-dependent branching, and in-place mutation hazards. Produce a verdict of READY, LIGHT REFACTOR, SIGNIFICANT REFACTOR, or NOT RECOMMENDED, with concrete refactor pointers.

Backend & APIsscripts

alibabacloud-data-agent-skill

Included

Invoke Alibaba Cloud Apsara Data Agent for Analytics via CLI to perform natural language-driven data analysis on enterprise databases. Data Agent for Analytics is an intelligent data analysis agent developed by Alibaba Cloud Database team for enterprise users. It automatically completes requirement analysis, data understanding, analysis insights, and report generation based on natural language descriptions. This tool supports: discovering data resources (instances/databases/tables) managed in DMS, initiating query or deep analysis sessions, real-time progress tracking, and retrieving analysis conclusions and generated reports. Use this Skill when users need to query databases, analyze data trends, generate data reports, ask questions in natural language, or mention "Data Agent", "data analysis", "database query", "SQL analysis", "data insights".

Backend & APIsscripts

token-optimizer

Included

Reduce OpenClaw token usage and API costs through smart model routing, heartbeat optimization, budget tracking, and native 2026.2.15 features (session pruning, bootstrap size limits, cache TTL alignment). Use when token costs are high, API rate limits are being hit, or hosting multiple agents at scale. The 4 executable scripts (context_optimizer, model_router, heartbeat_optimizer, token_tracker) are local-only — no network requests, no subprocess calls, no system modifications. Reference files (PROVIDERS.md, config-patches.json) document optional multi-provider strategies that require external API keys and network access if you choose to use them. See SECURITY.md for full breakdown.

Backend & APIsscripts

resend-cli

Included

Use this skill when the task is specifically about operating Resend from an AI agent, terminal session, or CI job via the official resend CLI: installing/authenticating the CLI, sending/listing/updating/cancelling emails, batch sends, domains and DNS, webhooks and local listeners, inbound receiving, contacts, topics, segments, broadcasts, templates, API keys, profiles, or debugging Resend CLI/API failures. Trigger on mentions of Resend CLI, `resend`, `resend doctor`, `resend emails send`, `resend domains`, `resend webhooks listen`, `resend emails receiving`, or agent-friendly terminal automation.

Backend & APIsscripts

alibabacloud-odps-maxframe-coding

Included

Use this skill for MaxFrame SDK development and documentation navigation on Alibaba Cloud MaxCompute (ODPS). Helps answer MaxFrame API, concept, official example, and supported pandas API questions; create data processing programs; read/write MaxCompute tables; debug jobs (remote or local); and build custom DPE runtime images. Trigger when users mention MaxFrame, MaxCompute with MaxFrame, ODPS table processing, DPE runtime, MaxFrame docs/examples, DataFrame/Tensor operations, or GPU runtime setup. Works for both English and Chinese queries about Alibaba Cloud data processing with MaxFrame.

Backend & APIsscripts