playground

What this skill does

# Prompt Playground

The prompt playground is a tool for authoring and optimizing prompts. It supports two different
ways of working: fast manual prompt iteration without a dataset, and dataset-backed prompt
experimentation with evaluators and experiments. Choose the workflow that matches the user's
current goal and the UI context they have mounted.

## Workflow: Create And Iterate Without A Dataset

Use this workflow when the user wants to draft, rewrite, or manually improve a prompt and no
dataset-backed evaluation loop is in scope.

1. Clarify the task the prompt must perform: input variables, expected output shape, audience,
   constraints, and examples of good or bad behavior when available.
2. If a playground prompt already exists, call `read_prompt_instance` before proposing changes so
   you have the current messages, message IDs, labels, and revision.
3. Draft or revise the prompt so it clearly states the task, required context, output contract, and
   success criteria. Keep the prompt directly tied to the user's stated goal.
4. Use `edit_prompt_instance` for changes to the mounted prompt so the user can review the diff
   before accepting it.
5. Use `add_prompt_instance` when the user wants a fresh comparison instance that starts from the
   default prompt messages. Use `clone_prompt_instance` when comparing alternatives should preserve
   existing prompt content as the starting point. Discuss variants by their alphabetic labels, but
   pass numeric instance IDs to tools. After adding, use the returned `addedInstance` snapshot for
   follow-up edits.
6. Use `set_variable_values` when the user provides manual values for prompt template variables.
7. Call `run_playground` only when the user asks to run, try, test, or compare the current prompt.
   Treat the output as qualitative feedback rather than dataset-backed evidence.
8. After the run finishes, call `read_playground_output` to inspect raw output and get the traceId
   for trace analysis when needed.
9. Call `save_prompt` only when the user explicitly asks to save or confirms that the current
   prompt should be persisted. For a first-time save of an unsaved prompt, omit `name` unless the
   user provided one; the tool will derive a valid Phoenix prompt name from the prompt content.
   Always pass a save description; it should read like a clear, short git commit message. Treat
   tags like releases and do not promote tags unless the user asks.
10. Inspect the output with the user, identify the next concrete improvement, and repeat the edit or
   comparison loop until the prompt is useful for the task.

## Workflow: Iterate Over A Dataset With Evaluators And Experiments

Use this workflow when the user wants evidence that a prompt is improving across a dataset, or when
they are comparing prompt variants using evaluator results.

1. Load the dataset with `load_dataset` if it isn't already loaded. If the user named a dataset but
   no split and the dataset has splits, name them and ask whether to scope to one or load the whole
   dataset — then load once.
2. Confirm the dataset represents the task the prompt is meant to solve, including the important
   input fields, expected outputs, and failure modes.
3. Make sure the starting prompt is well formed before running it: it should define the task,
   relevant variables, output format, and any constraints needed for consistent evaluation.
4. Use `set_playground_experiment_recording` before running when the user wants the next
   dataset-backed playground run recorded, persisted, or saved as an experiment. Set it to false
   only when the user explicitly asks for a temporary, throwaway, unrecorded, or ephemeral run.
   This is separate from `save_prompt`, which saves prompt versions rather than run results.
5. Run the playground over the dataset. When recording is enabled, each prompt instance run over a
   dataset is captured as an experiment, with outputs and evaluator annotations available for
   review.
6. Review the experiment outputs and annotations to find recurring failure patterns (see *Reading
   experiment results* below for the `phoenix-gql` query); `read_playground_output` only reads
   manual playground runs. Separate model randomness from prompt issues when possible.
7. Use or add evaluators when they make issue detection more systematic, especially for failures
   that are hard to spot by manual review alone.
8. Form a specific hypothesis for improving the prompt, then use `edit_prompt_instance`,
   `add_prompt_instance`, or `clone_prompt_instance` to create the next candidate. Choose
   `add_prompt_instance` for a candidate that starts from the default prompt messages and
   `clone_prompt_instance` for a candidate that should start from existing prompt content.
9. Rerun the playground and compare experiments. Look for evaluator improvements, fewer repeated
   failure modes, and acceptable tradeoffs in output quality.
10. Use `save_prompt` to save a prompt as a new version only after the evidence shows an
   improvement or the user explicitly accepts the tradeoff. For unsaved prompts, the tool can
   create the Phoenix prompt directly without asking for a name unless the user cares about the
   exact name.
11. Continue the hypothesis, edit, run, compare loop until the dataset-backed results satisfy the
   user's goal.

### Reading experiment results

When an instance carries an `experimentId`, read its cost and evaluator scores with `phoenix-gql`:

```
phoenix-gql --vars '{"experimentId":"<id>"}' 'query($experimentId: ID!){ node(id:$experimentId){ ...on Experiment { runCount expectedRunCount job{status} costSummary{total{cost tokens}} annotationSummaries{annotationName meanScore count errorCount} } } }'
```

An `experimentId` only means the experiment is queryable, not that the run finished — trust the
summaries as final only when `job.status` is `COMPLETED` or `runCount == expectedRunCount`. To
compare reruns, re-query earlier experiment IDs from the conversation and diff their summaries.
Experiments from unrecorded runs are ephemeral and the server sweeps them ~24h after their last
update; a freshly surfaced `experimentId` is well within that window, but an id re-queried from
much earlier in a long session may no longer resolve.

## Workflow: Author, Refine, Or Remove A Function Tool

Use this workflow when the user wants the model to be able to call a function/tool from the prompt,
when they want to refine the signature of an existing one, or when they want to remove a tool.
Function tools are JSON-Schema function definitions stored on the playground prompt instance
(alongside messages and model config). They are the things the model can "call" during a run.

1. Call `read_prompt_tools` before doing anything else. The result gives you the current tool list,
   each tool's id and kind, and a `revision` token. Use the existing ids and names to decide
   whether you should update an existing tool, create a new one, or delete one.
2. If the user described a function in words, propose a concrete JSON Schema for it. Default to
   lowercase snake_case parameter names and a `{"type":"object","properties":{...},"required":[...]}`
   shape unless the user specifies otherwise.
3. Call `write_prompt_tools` with the latest `revision`. Put every change in a single call: `tools`
   is an array of creates/updates (omit `id` to create, pass an existing `id` to patch — only the
   fields you include change), and `deleteToolIds` is a list of ids to remove. Deletes may target
   `raw` vendor tools too, even though writes can't. The batch is all-or-nothing: if any change is
   invalid (missing id, a `raw` tool on the write path, or the same id created/updated and deleted)
   nothing is applied and the error explains which. Deleting the tool that is the forced tool choice
   is allowed — the choice is reset to auto and reported back; mention that to the user.
4. After the write, briefly summarize what changed in plain English (which tools were