agent-platform-eval-flywheel

What this skill does

# Agent Platform Eval Flywheel Skill

Help users evaluate and iteratively improve GenAI models and agents using
the Agent Platform GenAI Evaluation SDK (`google.genai` / `agentplatform`).

## When to use this skill

-   Evaluating GenAI agents or models with the Agent Platform GenAI
    Evaluation SDK (`client.evals.evaluate()`).
-   Creating evaluation datasets from session traces, pandas DataFrames, or
    synthetic generation.
-   Selecting, configuring, or writing custom evaluation metrics.
-   Analyzing rubric verdicts, loss patterns, and clustering failures.
-   Suggesting concrete code/prompt improvements based on eval results.

## Setup

Install the SDK:

```bash
pip install google-cloud-aiplatform[evaluation]>=1.154.0 google-genai>=1.0.0
```

Need `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION`. Check env vars
first; if missing, ask the user. Newer Gemini models often need
`location="global"`.

## The Quality Flywheel

Five stages, run in order on the first pass, then loop 2 → 5 until quality
targets are met.

### Shortcuts that waste time

| Shortcut                                                 | Why it fails                                                                                              |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| "I'll tune the metric threshold down so it passes."      | Hides real failures. Fix the agent, not the bar.                                                          |
| "This case is flaky, I'll skip it."                      | Flakiness reveals non-determinism in the agent. Fix with `temperature=0` or stricter instructions.        |
| "I just need to fix the eval dataset, not the agent."    | If expected outputs keep moving, the agent has a behavior problem.                                        |
| "I can tell from the trace it works — skip Stage 3."     | Self-grading doesn't generalize. Always run `evaluate()` and read scores.                                 |
| "One iteration is enough."                               | Expect 5–10+ iterations. Stopping early leaves regressions on other metrics undetected.                   |

### 1. Prepare Data

Produce an `EvaluationDataset`. There are three input shapes, pick the one
that matches the data the user already has:

-   **`EvalCase` list (single-turn or multi-turn):**

    ```python
    from agentplatform import types
    dataset = types.EvaluationDataset(eval_cases=[
        types.EvalCase(prompt="What is 2+2?", response="4", reference="4"),
        # For multi-turn agent traces, set agent_data instead of prompt/response.
    ])
    ```

    Multi-turn agent traces wrap each conversation in `AgentData` →
    `ConversationTurn` → `AgentEvent`. See
    [references/dataset_schema.md](references/dataset_schema.md) for the
    full type hierarchy.

-   **Pandas DataFrame (tabular sources — CSV, BigQuery, Sheets):**

    ```python
    import pandas as pd
    from agentplatform import types

    df = pd.DataFrame({
        "prompt":    ["What is 2+2?", "Capital of France?"],
        "response":  ["4",            "Paris"],
        "reference": ["4",            "Paris"],
    })
    dataset = types.EvaluationDataset(eval_dataset_df=df)
    ```

    Column names must match the fields the chosen metrics expect (see
    [references/dataset_schema.md](references/dataset_schema.md) for the
    per-metric requirements table).

-   **Cold start (no data at all):** synthesize scenarios server-side with
    `client.evals.generate_user_scenarios(...)` and a
    `UserScenarioGenerationConfig` (`user_scenario_count`,
    `simulation_instruction`, `environment_data`). Stage 2 plays them out.

For ADK session dumps, use `scripts/parse_adk_traces.py` instead of writing
the conversion by hand.

### 2. Run Inference

Populate responses/traces on the dataset. **Skip this stage** if traces are
already complete (e.g., production logs or replay).

```python
# Agent eval — pass a callable wrapping the user's ADK Agent/App.
client.evals.run_inference(model=agent_callable, src=dataset)

# Model eval — pass a model ID directly.
client.evals.run_inference(model="gemini-2.5-flash", src=dataset)

# Synthesized scenarios — let the simulator drive.
client.evals.run_inference(
    model=agent_callable,
    src=dataset,
    user_simulator_config=UserSimulatorConfig(max_turn=10),
)

# DataFrame also works as src= — no EvalCase wrapping needed.
client.evals.run_inference(model="gemini-2.5-flash", src=df)
```

### 3. Grade (always run)

```python
result = client.evals.evaluate(dataset=dataset, metrics=[...])
```

**Pick metrics by what you want to measure.** Full catalog in
[references/metric_registry.md](references/metric_registry.md).

**Agent metrics (multi-turn, adaptive rubrics)** — start here for agent eval.

| Goal                                          | Metric                          |
| --------------------------------------------- | ------------------------------- |
| Did the agent achieve the user's goal?        | `multi_turn_task_success`       |
| Was the reasoning path logical and efficient? | `multi_turn_trajectory_quality` |
| Tool/function calling quality across turns    | `multi_turn_tool_use_quality`   |
| Overall conversational quality                | `multi_turn_general_quality`    |
| Final response quality (no reference needed)  | `final_response_quality`        |
| Final response vs. a golden reference         | `final_response_match`          |
| Single-turn tool use                          | `tool_use_quality`              |

**General quality metrics (single-turn, adaptive rubrics)** — for model eval.

| Goal                                                  | Metric                  |
| ----------------------------------------------------- | ----------------------- |
| Overall response quality (recommended starting point) | `general_quality`       |
| Linguistic quality (fluency, coherence, grammar)      | `text_quality`          |
| Adherence to specific constraints / instructions      | `instruction_following` |

**Static rubric metrics (fixed criteria)** — apply alongside the above.

| Goal                                              | Metric          |
| ------------------------------------------------- | --------------- |
| Catch hallucinated claims (RAG, factual answers)  | `hallucination` |
| Factuality / consistency against provided context | `grounding`     |
| Safety policy compliance                          | `safety`        |

**Domain-specific check no built-in covers:** write a custom metric.

-   **Predefined:** `types.RubricMetric.<NAME>` — server-side AutoRater, no
    judge model needed.
-   **Custom LLM-as-a-judge:** `types.LLMMetric` with `prompt_template` or
    `types.MetricPromptBuilder` for structured rubrics.
-   **Custom code:** `types.CodeExecutionMetric` with a `custom_function`
    string containing `def evaluate(instance: dict)` for remote sandboxed
    execution; or `types.Metric` with `custom_function=<callable>` for
    local execution.

**Always persist the result** so Stage 4 and 5 can read it. Save both JSON
(machine-readable, diffable) and HTML (human-readable, linkable):

```python
import datetime
from pathlib import Path

from agentplatform._genai import _evals_visualization

out_dir = Path("artifacts/grade_results")
out_dir.mkdir(parents=True, exist_ok=True)
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")

result_json = result.model_dump_json()
(out_dir / f"results_{ts}.json").write_text(result_json)

html = _evals_visualization.get_evaluation_html(result_json)
(out_dir / f"results_{ts}.html").write_text(str(html))
```

Or after the fact: `scripts/render_html_report.py --type evaluation` or
`scripts/inspect_results.py --save-html`.

### 4. Analyze Failures

Read `summary_metrics` and `eval_case_results` — never fabricate scores.
Use `scripts/inspect_results.py --failing-only