agent-platform-tuning

What this skill does

# Agent Platform Model Tuning

## Overview

This skill provides procedural knowledge for fine-tuning Large Language Models
(both Open Models and Gemini Models) using Agent Platform's tuning service. It
covers the entire lifecycle from environment setup and data preparation to job
configuration, monitoring, and deployment.

## Workflow Decision Tree

1.  **Model Category Identification**: Has the user explicitly stated whether
    they want to tune an **Open Model** or a **Gemini Model**?

    -   **No** → **STOP**. Ask the user if they want to tune an Open Model or a
        Gemini Model. **CRITICAL EXCEPTION for Environment Setup Requests:** If
        the user is specifically asking for environment setup instructions
        (e.g. "What environment setup is needed?"), you **MUST** provide the full
        [Phase 0 environment setup](#phase-0) instructions in your initial
        response, *simultaneously* with asking clarifying questions about the
        model category.
    -   If the user provides a specific tuning purpose, you should
        recommend three models: one Open Model, one Gemini Model, and a third
        generally recommended choice. Briefly list the pros and cons of each
        (e.g., Gemini models might be more expensive, etc.). **CRITICAL:** You
        must read `references/models.md` during this step and only recommend
        models explicitly listed in that catalog. Do not recommend unsupported
        models like Mistral. Do not proceed with model configuration until the
        category is confirmed.
    -   **Yes** → Proceed.

2.  **Environment Check**: Has the environment (Auth, APIs, IAM, Venv) been
    initialized?

    -   **No** → Go to [Phase 0: Environment & IAM Setup](#phase-0).
    -   **Yes** → Proceed.

3.  **Dataset Status**: Is the dataset ready in JSONL format, **is its structure
    valid for tuning**, and is it uploaded to Google Cloud Storage?

        -   **No** → Go to [Phase 1: Dataset Preparation & Upload](#phase-1).
        -   **Yes** → Proceed.

4.  **Column Selection Confirmation**: Have you presented the columns to the
    user and confirmed the mapping?

    -   **No** → **STOP**. You must show samples and get user confirmation on
        column mapping as described in Phase 1.0 before proceeding.
    -   **Yes** → Proceed.

5.  **Configuration**: Has the user provided the target model and
    hyperparameters, or explicitly agreed to your recommendations?

    -   **No** → Go to
        [Phase 2: Model Configuration & Recommendation](#phase-2).
    -   **Yes** → Proceed.

6.  **Job Status**: Has the tuning job been submitted?

        -   **No** → Go to
            [Phase 3: Tuning Job Execution](#phase-3-tuning-job-execution).
        -   **Yes** → Proceed.

7.  **Job Completion**: Is the tuning job complete?

        -   **No** → Go to [Phase 4: Monitoring](#phase-4-monitoring).
        -   **Yes** → Proceed.

8.  **Deployment**: Has the tuned model been deployed (if required)?

        -   **No** → Go to [Phase 5: Model Deployment](#phase-5-model-deployment).
        -   **Yes** → Task Complete.

## Phase 0: Environment & IAM Setup {#phase-0}

Ensure the foundational environment is ready before proceeding.

### 0.1 Authentication & Project Context

-   Check if `gcloud` CLI is installed. If it is not installed, prompt the
    user for permission to install it before proceeding. If it is installed,
    update it:

```bash
gcloud components update --quiet > /dev/null 2>&1
```

-   Verify `gcloud auth list`. If not authenticated, run `gcloud auth login`.
-   Ensure `project` and `location` are known. Use `gcloud config get project`
    to retrieve the current project (and `gcloud config get compute/region` for
    region).
-   **CRITICAL: Ask for Confirmation.** You must prompt the user to confirm the
    retrieved project and region before proceeding, in case they want to switch
    to a different one.

### 0.2 Possible Locations

The following locations are available for tuning:

-   us-central1
-   europe-west4
-   us-west1
-   us-east5
-   asia-southeast1

No other values are supported for this section, ensure that the location is
listed above.

### 0.3 Enable APIs

Ensure `aiplatform.googleapis.com` and `storage.googleapis.com` are enabled.

```bash
gcloud services enable aiplatform.googleapis.com storage.googleapis.com \
    --project=YOUR_PROJECT
```

### 0.4 IAM Permissions

Verify the following identities have the required roles.

-   **Agent Platform Service Agent**:
    `[email protected]`
-   **Managed OSS Fine Tuning Service Agent**:
    `service-PROJECT_NUMBER@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com`
-   **User Identity**: The account running the commands.

### 0.5 Virtual Environment

Create and use a virtual environment named `tuning_agent_venv` in the home
directory. Install dependencies from `references/requirements.txt`.

```bash
python3 -m venv ~/tuning_agent_venv
source ~/tuning_agent_venv/bin/activate
pip install -r references/requirements.txt
```

**CRITICAL AGENT INSTRUCTION:** You **MUST** ensure that every Python command or
script execution (e.g., `python3 scripts/...`, `pip install ...`) is prefixed
with the virtual environment activation command: `source ~/tuning_agent_venv/bin/activate &&`.
Additionally, advise the user that every single time they run a Python command,
execute a script, or inspect data inline, they **MUST** also activate this
virtual environment first in their bash execution. For example:
`source ~/tuning_agent_venv/bin/activate && python3 ...`.
Do not run standalone `python3` commands without activating the environment, as
they will encounter `ModuleNotFoundError` issues.

## Phase 1: Dataset Preparation & Upload {#phase-1}

### 1.0 Dataset Discovery & Confirmation

-   **User-Provided Dataset Verification:** If the user specifies a dataset
    filename or path in their prompt, verify its existence in the workspace
    (e.g. via script execution or checking for typos).
    *   **If the file cannot be found anywhere**, you **MUST** inform the user
    that the dataset file does not exist or cannot be accessed. You **MUST**
    prompt the user to provide a valid dataset path. Alternatively, if candidate
    dataset files are found in the workspace during your search, you **MUST**
    present the candidates to the user and ask them to select one. You **MUST**
    stop tool execution immediately after reporting the missing file or
    presenting candidates, and wait for the user's response. Do **NOT** ask for
    80/20 validation split permission, and do **NOT** attempt to upload the
    dataset before receiving a valid dataset file selection from the user.
    *   **If the file is found and verified**, proceed to Step 1.1 Formatting &
    Validation below.
-   **Auto-Discovery: From User Bucket:** If the user does not have a dataset
    and no suitable alternative is found in the Hugging Face reference, offer to
    search the user's GCS buckets for potential training data. Prioritize
    searching for files with extensions like `.jsonl`, `.json`, `.csv`, and
    `.parquet`. If such files are found, read the first few lines/records of
    each to determine if they contain text-based data suitable for tuning
    (e.g., prompt/completion pairs) that can be modified to follow
    [Data Preparation Guide](references/data_prep.md) and is related to the
    tuning task requested. **DO NOT** search without prompting first.
-   **Auto-Discovery: From Task to Huggingface:** If the user has a specific
    task, refer to [Huggingface Datasets Reference](references/hf_datasets.md)
    and recommend a dataset from this if one exists. For each dataset
    recommended, provide some information about the dataset and provide some
    reasonable splits. > [!IMPORTANT] > **CRITICAL: Ask for Confirmation and
    Column Selection.** Do not proceed > with dataset preparation or upload
    until you perform the followi