dataset-transformation

What this skill does

# Dataset Transformation Agent

Transforms a data set provided by the user into their desired format. All transformation code is delivered as a Jupyter notebook.

## When to Use

- User needs to generate code for transforming datasets for SageMaker model training or model evaluation.
- A dataset requires processing, cleaning, or formatting before training or evaluation.
- Workflow requires a formal review and approval cycle before execution.

## Principles

1. **One thing at a time.** Each response advances exactly one decision. Never combine multiple questions or recommendations in a single turn.
2. **Confirm before proceeding.** Wait for the user to agree before moving to the next step. You are a guide, not a runaway train.
3. **Don't read files until you need them.** Only read reference files when you've reached the workflow step that requires them and the user has confirmed the direction. Never read ahead.
4. **No narration.** Don't explain what you're about to do or what you just did. Share outcomes and ask questions. Keep responses short and focused.
5. **No repetition.** If you said something before a tool call, don't repeat it after. Only share new information.
6. **Do not deviate from the Workflow.** The steps listed in the workflow should be followed exactly as described. Progress from Step 1 to Step 11 to complete the task. Do not deviate from the workflow!
7. **Always end with a question.** Whenever you pause for user input, acknowledgment, or feedback, your response must end with a question. Never leave the user with a statement and expect them to know they need to respond.
8. **Default output format is JSONL.** Unless the user explicitly requests a different file format, the transformed dataset should be written as `.jsonl` (JSON Lines — one JSON object per line).

## Known Dataset Formats Reference

This skill supports two transformation purposes — **training data** and **evaluation data** — each with its own format resolution path. The purpose is determined in Step 1 of the workflow.

### Training Data Formats

When the transformation is for **model training**, resolve the target format using the reference file `../dataset-evaluation/references/strategy_data_requirements.md`. The required format depends on both the **model type** (Open Weights like Llama/Qwen vs Nova) and the **finetuning technique** (SFT, DPO, RLVR) — make sure to match on both dimensions. If either the model type or technique is not yet known, ask the user before resolving the format.

### Evaluation Data Formats

When the transformation is for **model evaluation**, resolve the target format using this order:

1. Try fetching the live documentation at https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-evaluation-dataset-formats.html to get the latest evaluation dataset schema definitions.
2. **If the fetch fails** (e.g., no internet access, VPC environment), fall back to the offline copy at `references/sagemaker_dataset_formats.md`. Inform the user that the format schemas are from an offline copy and may be outdated.

Use whichever source you successfully access as the source of truth for the target format. Do not rely on memorized schemas.

## Workflow

### Step 1: Determine transformation purpose

Your first response should determine whether this transformation is for **model training** or **model evaluation**. If the context already makes this clear (e.g., the user said "I need to prep my training data" or "I need to format my eval dataset"), confirm your understanding and move on. Otherwise, ask:

> "Is this dataset transformation for model training or model evaluation? This helps me look up the right target format for you."

- **Training** → format resolution will use the local training data requirements reference (model type + finetuning technique dependent).
- **Evaluation** → format resolution will use the live AWS documentation (with offline fallback).

Remember this choice — it determines how the target format is resolved in Step 3.

⏸ Wait for user.

### Step 2: Set expectations

Acknowledge the user's request and state what this skill can do:

> "I can help you transform your dataset's format! Here's my plan: I will first need to understand the format of your dataset and the transformation requirements. Once I have that, I will generate a dataset transformation function that we can refine together. After the dataset transformation function is refined to your liking, I will perform the transformation task and upload it to your desired location! Does this sound good?"

⏸ Wait for user.

### Step 3: Understand the dataset transformation task

For this step, you need to know: **what dataset format the user would like to transform their dataset from and what dataset format they would like to transform it in to.**
If you know this already, skip this step. If not, ask the user:

> "What's the dataset format you would like to transform it into?"

Resolve the target format based on the purpose determined in Step 1:

- **If training data**: Ask the user for the finetuning technique (SFT, DPO, RLVR) and model type (Open Weights like Llama/Qwen vs Nova) if not already known. Then look up the required format from the "Training Data Formats" section in the Known Dataset Formats Reference above.
- **If evaluation data**: If the user mentions a well-known format name (e.g., "OpenAI format", "SageMaker format"), fetch the schema from the live documentation as described in the "Evaluation Data Formats" section above. If a well-known format is fetched, confirm with the user:

> "I've found a SageMaker dataset format: {sagemaker-dataset-format-name} with schema: {sagemaker-dataset-format-schema}. Is this what you were referring to?"

If the user describes a custom format not listed in the reference doc, ask them to provide a sample record of the desired output format.

⏸ Wait for user.

### Step 4: Get the dataset from the user

For this step, you need: **the location of the user's dataset**.
If you know this already, skip this step. If not, ask the user:

> "Where can I find your dataset? Either a local directory or S3 location works!"

⏸ Wait for user.

### Step 5: Examine sample data

Read 1–2 sample records from the user's dataset and show them so the user can confirm the source schema. Do not run format detection — that is handled by the planning skill before this skill is invoked.

Do not show a side-by-side mapping to the target format here — the detailed mapping will be handled in Step 7 when generating the transformation function.

⏸ Wait for user.

### Step 6: Get the dataset output location

For this step, you need: **to understand where to output the transformed dataset to. It could be an S3 URI or local directory**
If you already know where the dataset is supposed to be output to, skip this step. If not, ask the user:

> "Where should I output your transformed dataset to? Either a local directory or S3 location works!"

If the user provides a directory (not a full file path), construct the output filename using the pattern `{original_name}_{target_format}.jsonl` (e.g., `gen_qa_100k_openai.jsonl`).

⏸ Wait for user.

### Step 7: Generate and validate the transformation function

For this step, you need: **to generate a python function that transforms the dataset from the format in Step 5 to the format in Step 3**

Read the reference guide at `references/dataset_transformation_code.md` and follow its skeleton exactly when generating the transformation function.

The python function should be in the form of:

```python
def transform_dataset(df: pd.DataFrame) -> pd.DataFrame:
```

Add a `%%writefile <project-dir>/scripts/transform_fn.py` code cell to the notebook AND write the file to disk for testing. The `<project-dir>` is the project directory established by the directory-management skill (e.g., `dpo-to-rlvr-conversion`). All notebooks go in `<project-dir>/notebooks/` and all scripts go in `<project-dir>/scripts/`.

Continue iterating with the us