llm-obs-experiment-py-bootstrap

What this skill does

# LLM Obs Experiment (Python) Bootstrap — Generate a Python Experiment Using `ddtrace.llmobs`

Produce a single self-contained Python experiment that uses the official **`ddtrace.llmobs` SDK**. Output is either a `.py` script or an `.ipynb` notebook. The generated code mirrors the patterns shown in DataDog's reference notebooks at <https://github.com/DataDog/llm-observability/tree/main/experiments/notebooks>.

The SDK handles lazy project/experiment creation, dataset push diffing, the 5 MB / 1000-record bulk threshold, eval metric streaming, and the status state machine on the user's behalf. This skill must therefore **never re-implement those primitives** — it just imports `LLMObs` and trusts it.

## Usage

```
/llm-obs-experiment-py-bootstrap [--format py|ipynb] [--dataset <path>] [--dataset-name <name>] [--dataset-version <int>] [--project-name <name>] [--evaluator-style function|class|remote] [--jobs <n>] [--output <path>]
```

Arguments: $ARGUMENTS

### Inputs

All inputs are optional. If the user omits a flag, fall back to the default — never block on prompting for `--jobs`, `--format`, etc.

| Input | Default | Description |
|---|---|---|
| `--format` | `py` | `py` (single `.py` file) or `ipynb` (Jupyter notebook with one cell per section). |
| `--dataset` | none — emit a sample 3-record `records=[...]` inline so the file is runnable as-is | Path to a local `DatasetRecordRaw[]` JSON or CSV. JSON → `create_dataset(records=...)`; CSV → `create_dataset_from_csv(...)`. Mutually exclusive with `--dataset-name`. |
| `--dataset-name` | none | Name of an existing Datadog dataset to fetch at runtime via `LLMObs.pull_dataset(...)`. Use this when the dataset already lives in Datadog (e.g. created in the UI or by a prior run) — no local file required. Mutually exclusive with `--dataset`. |
| `--dataset-version` | none (latest) | Pin to a specific dataset version when using `--dataset-name`. Passed through as `pull_dataset(version=N)`. Ignored if `--dataset-name` is not set. |
| `--project-name` | `experiment-<service-name>` — derived from the codebase (see Workflow step 1); falls back to `experiment-sdk-default` only if nothing resolves | Datadog project name (visible in the LLM Experiments UI). The SDK's `ml_app` tag falls back to this automatically — no separate flag needed. |
| `--evaluator-style` | `function` | `function` (plain functions — notebook default), `class` (`BaseEvaluator` subclasses), or `remote` (`RemoteEvaluator` instances). |
| `--jobs` | `10` | Passed to `experiment.run(jobs=N)`. |
| `--output` | `./experiments/experiment.<ext>` | File extension derives from `--format`: `.py` or `.ipynb`. |

---

## SDK Surface (Cited)

These are the public symbols the generated code uses. All come from `ddtrace.llmobs` (the public package — never from `ddtrace.llmobs._experiment` or other underscore-prefixed modules).

| Import | Source | What it gives you |
|---|---|---|
| `LLMObs` | `ddtrace/llmobs/__init__.py` re-exports `_llmobs.py` | `.enable()`, `.create_dataset()`, `.create_dataset_from_csv()`, `.pull_dataset(dataset_name, project_name, version)`, `.experiment()`, `.async_experiment()` |
| `RemoteEvaluator`, `EvaluatorContext` | `ddtrace/llmobs/__init__.py` | LLM-as-Judge that runs server-side; preferred over inline `LLMJudge` |
| `BaseEvaluator`, `EvaluatorResult` | `ddtrace/llmobs/__init__.py` | Class-based evaluator path (advanced) |
| `LLMJudge` | `ddtrace/llmobs/_evaluators/llm_judge.py` (re-exported) | Inline LLM-as-Judge with prompt template support |

**Canonical call signatures** (must match the generated code exactly):

```python
LLMObs.enable(
    api_key=os.getenv("DD_API_KEY"),
    app_key=os.getenv("DD_APPLICATION_KEY"),
    site=os.getenv("DD_SITE", "datadoghq.com"),  # required for non-prod sites (e.g. datad0g.com, datadoghq.eu)
    project_name="<project>",
    agentless_enabled=True,  # required when not running behind the dd-agent
)
# Note: ml_app is not a separate input. The SDK derives it from project_name
# when not supplied. If a user really wants to override it later, they can
# add `ml_app="..."` to enable() themselves.

dataset = LLMObs.create_dataset(
    dataset_name="<name>",
    description="<optional>",
    records=[
        # Per-record `tags` MUST be a list of "key:value" strings (e.g. "env:smoke"),
        # never bare strings — the SDK rejects malformed tags with a ValueError on append.
        {"input_data": {"<k>": "<v>"}, "expected_output": "<v>", "metadata": {}, "tags": ["env:<env>"]},
        # ...
    ],
)
# OR
dataset = LLMObs.create_dataset_from_csv(
    csv_path="<path>",
    dataset_name="<name>",
    input_data_columns=["<col1>", "<col2>"],
    expected_output_columns=["<col>"],
)
# OR pull an existing Datadog dataset by name (no local file needed)
dataset = LLMObs.pull_dataset(
    dataset_name="<name>",
    project_name="<project>",   # optional — defaults to the project on enable()
    version=2,                  # optional — pin a version; omit for the latest
)

def task_fn(input_data: dict, config: dict):
    # TODO(user): replace with your actual LLM call
    ...

# Plain function evaluator (default style)
def exact_match(input_data, output_data, expected_output) -> bool:
    return output_data == expected_output

experiment = LLMObs.experiment(
    name="<experiment_name>",
    dataset=dataset,
    task=task_fn,
    evaluators=[exact_match],
    config={
        "model": "gpt-4o-mini",
        "temperature": 0.0,
        # Provenance also lives in `config` so it renders in the
        # experiment's Configuration view alongside model/temperature.
        # `tags=` below only reaches metadata.tags, which the current UI
        # does not surface as chips — config is what users actually see.
        "generated_by": "claude-code",
        "skill": "llm-obs-experiment-py-bootstrap",
    },
    description="<optional>",
    tags={
        # Same provenance, sent to experiment metadata.tags for any future
        # tag-filter UI / API consumers. Always emitted alongside the
        # config copy — never one without the other.
        "generated_by": "claude-code",
        "skill": "llm-obs-experiment-py-bootstrap",
    },
)

experiment.run(jobs=10)
print(experiment.url)
```

---

## Evaluator Styles

Generated code uses **one** of three evaluator surfaces, picked by `--evaluator-style`. Whichever style is chosen, **prefer returning `EvaluatorResult` over a bare `bool`/`float`** whenever the evaluator has any signal beyond the raw value — see "Return EvaluatorResult, not bare values" below.

### Return `EvaluatorResult`, not bare values

Plain functions are allowed to return `bool` / `float` / `dict`, and `BaseEvaluator.evaluate()` is allowed to return raw `JSONType`. The SDK accepts both — but `EvaluatorResult` carries fields the Datadog UI surfaces in ways the raw value cannot:

| Field | Type | Used by Datadog UI for |
|---|---|---|
| `value` | `bool` / `float` / `str` / `dict` (JSONType) | The score itself — shown on the experiment metric. **Required.** |
| `reasoning` | `str` | Per-record explanation shown in the compare UI; lets reviewers see *why* an evaluator passed/failed without re-running the LLM. |
| `assessment` | `str` (e.g. `"pass"` / `"fail"` / `"partial"`) | Determines whether a metric trend going up vs. down is an improvement; the UI uses this to color baseline-vs-candidate comparisons. |
| `metadata` | `dict[str, JSONType]` | Free-form per-record context (e.g. `{"confidence": 0.95}`); shown in record drill-down. |
| `tags` | `dict[str, JSONType]` | Used to slice experiment results in the UI (e.g. `{"category": "accuracy"}`). |

The generated code should default to `EvaluatorResult` for any evaluator richer than a one-line equality check. The trivial `exact_match` and `length_under_500` shown below are the only cases where a bare `bool` is acceptable.

### `function` (default — what the notebooks use)

Plain Python functions with the signature `(input_data, output_data, expected