datarobot-agent-assist

What this skill does

# DataRobot Agent Assist

This skill merges **agent design, coding, and deployment** with **interactive dress-rehearsal simulation** in one place.

Assistance falls into three categories:

1. **Designing an AI agent** → Clarify requirements, build `agent_spec.md`, optionally simulate the agent before coding
2. **Coding an AI agent** → Adapt the DataRobot agent application template to the spec
3. **Deploying an AI agent** → Follow `AGENTS.md` deployment instructions

If the user's first message is simply `1`, `2`, or `3`, treat it as selecting one of these categories.

---

## On Activation

Present the three options clearly:

```
Welcome! I help you design, code, and deploy AI agents (with optional dress-rehearsal simulation before coding).

What would you like to do?
  1. Design an AI agent     → Describe your idea
  2. Code an AI agent       → Load and implement an existing agent_spec.md
  3. Deploy an AI agent     → Deploy an implemented agent to DataRobot
```

Show this menu first. After the user selects an option (`1`, `2`, or `3`), run the **[Pre-requisite Check](#pre-requisite-check)** and then the **[Script Path Resolution](#script-path-resolution)** before doing anything else for that option.

---

## Script Path Resolution

Before invoking any helper script, resolve `<skill_scripts_dir>` once for the session:

- `<skill_scripts_dir>` is the `scripts/` subdirectory of the directory containing this `SKILL.md` file.
- Confirm it exists with `ls <path_to_this_skill_dir>/scripts/`. If the directory is missing, tell the user the skill installation is incomplete and stop.
- Use the resolved absolute path for every `<skill_scripts_dir>/...` reference in this skill.

---

## Pre-requisite Check

Run in order before proceeding:

1. **Git** — run `git --version`. If missing, tell the user to install from https://git-scm.com and stop.
2. **Python** — run `python --version`. If missing or below 3.11, tell the user to install Python 3.11+ from https://python.org and stop.
3. **DataRobot CLI** — follow **DataRobot CLI Setup** at the bottom:
   - If missing, **ALWAYS RUN** the install command before proceeding
   - **ALWAYS RUN** the upgrade command before proceeding
   - If not authenticated, **ALWAYS RUN** the auth command before proceeding

---

## 1. Designing an AI Agent

### Clarification Phase

- Ask **at most 2 rounds** of clarifying questions before proposing an initial draft spec. If tools are still ambiguous after two rounds, start simple.
- Focus questions on:
  - What the agent does and who uses it
  - What tools it needs and what external services those tools call
  - Whether those services require authentication (API key, OAuth2, bearer token, etc.)
  - Whether the user needs a custom frontend beyond the default chat UI

- If the user mentions UI-related needs early ("dashboard", "visualization", "multi-page", "admin panel", "settings page"), capture it immediately in the `frontend` field — do **not** defer.

### Model Selection

- To check available models: Run the helper script:
   ```
   python <skill_scripts_dir>/list_llm_models.py \
     --json
   ```

  **CRITICAL**: In case the script fails due to any reason, do **not** proceed. Instead, return the error message to the user and ask how they want to proceed.

- Recommend a `gpt-5`, `claude-4-5`, or `gemini-2.5` model from the list unless the user specifies cost or other constraints.
- If none of those preferred families appear in the catalog, pick the highest-capability available model by name — prefer ones containing `large`, `pro`, `opus`, or `sonnet` over `mini`, `haiku`, or `flash`.
- Only display the full model catalog when the user **explicitly** asks to browse models.
- If the user's desired model is unavailable, suggest starting with an available one and updating after implementation.

### Spec Display

- **Always write the current spec to `agent_spec.md`** (YAML format) whenever showing it to the user.
- Show the spec frequently and iteratively — even if incomplete or partial.
- Do **not** summarize the spec in prose; display it as YAML in a code block.
- After displaying, invite the user to refine system prompts, add/modify tools, change the model, or update examples.

### Frontend Check (Mandatory Before Coding or Simulating)

Before offering to simulate or code, if the spec does not already have a `frontend` field set, **always ask**:

> "The template includes a default chat UI — is that sufficient, or would you like a custom frontend such as a dashboard, data visualization, or multi-page app?"

Then update the spec accordingly:
- Default UI → `frontend.type: "chat"`
- Custom UI → `frontend.type: "multi-page"` or `"custom"` with `pages` and optional `requirements`

### Agent Simulation (Before Coding)

Before transitioning to coding, ask the user (exact wording):

> "Would you like to run a dress rehearsal simulation first? (recommended)"

Wait for their reply. If they say yes, follow **[Dress Rehearsal](#dress-rehearsal)** end to end: initialize with `rehearsal.py --init`, drive turns with `--session`, handle `NOTE:` / `DONE`, and produce the feedback report. Do not substitute improvised role-play or manual mock tool traces. If they decline or skip, proceed directly to coding — do not simulate without explicit confirmation.

Script path: `python <skill_scripts_dir>/rehearsal.py ...`

---

## Dress Rehearsal

Simulate an `agent_spec.md` interactively before writing any code. Responses go through the DataRobot LLM Gateway; the rehearsal script handles API calls, state, and output. You orchestrate the loop, handle out-of-character commands, and produce the feedback report at the end.

**Engine location:** `<skill_scripts_dir>/rehearsal.py` (relative to repository root).

### Step 1 — Initialize the session

```bash
python <skill_scripts_dir>/rehearsal.py --init [--spec agent_spec.md]
```

If `agent_spec.md` does not exist and no path was provided, say so and stop.

The script creates a unique session directory in the system temp dir and prints two lines:
```
session=<session_dir>
output=<output_file>
```

Retain `session_dir` for all subsequent calls. Read the `output_file` and display its contents verbatim, then say:

> You are now the **end user** of this agent. Type messages as a real user would.
>
> **Out-of-character commands:**
> - `NOTE: <text>` — record a design observation
> - `DONE` — end the session and generate your feedback report

### Step 2 — Simulation loop

Keep track of any notes and the number of turns as the session progresses — you'll need these for the report.

**On each user message:**

- If it starts with `NOTE:` — acknowledge the note, prompt for next message. Do not call the script.
- If it is `DONE` — proceed to Step 3.
- Otherwise — run the turn:

```bash
python <skill_scripts_dir>/rehearsal.py --session {session_dir} "{user_message}"
```

The script prints `output=<output_file>`. Read that file and display its contents verbatim. It will contain `[TOOL CALL]`, `[SIMULATED RETURN]`, and `[Agent]:` blocks as appropriate.

If the script exits non-zero, display the error and ask whether to continue or abort.

### Step 3 — Feedback report

Before writing the report, review the session and consider each of these areas — only surface the ones where you have something concrete to say:

- **System prompt** — wording, missing constraints, persona, tone
- **Tools** — input/output scoping, missing or redundant tools, argument naming
- **Model** — only flag if clearly wrong for the observed task complexity
- **Example prompts** — additions or revisions based on what was tested
- **Other** — edge cases, UX concerns, data dependency risks

Then write the report in this format:

```
════════════════════════════════════════════
  DRESS REHEARSAL REPORT
════════════════════════════════════════════

{1–2 sentences: what was tested and how the agent performed overall}
{If notes were recorded: "Notes: " followed by each note on its own line, prefixed with —}

Sugges