slime-user

What this skill does

# SLIME User Guide

SLIME is an LLM post-training framework for RL Scaling developed by THUDM. It supports various RL algorithms (GRPO, GSPO, PPO, Reinforce++), multiple training backends (Megatron, FSDP), and advanced features like multi-turn interactions, tool calling, and dynamic sampling.

## Quick Start Workflow

### For First-Time Users

1. **Environment Setup**
   - Use Docker: `docker pull slimerl/slime:latest`
   - Or build from source: See `docs/en/get_started/quick_start.md`
   - Hardware: Supports H100/H200, B200 series

2. **Download Model and Data**
   ```bash
   hf download Qwen/Qwen3-4B --local-dir /root/Qwen3-4B
   hf download --repo-type dataset zhuzilin/dapo-math-17k --local-dir /root/dapo-math-17k
   ```

3. **Convert Weights** (Megatron backend only)
   ```bash
   source scripts/models/qwen3-4B.sh
   PYTHONPATH=/root/Megatron-LM python tools/convert_hf_to_torch_dist.py \
       ${MODEL_ARGS[@]} \
       --hf-checkpoint /root/Qwen3-4B \
       --save /root/Qwen3-4B_torch_dist
   ```

4. **Run Training**
   ```bash
   bash scripts/run-qwen3-4B.sh
   ```

### For Experienced Users

When user needs specific functionality:
- **Multi-turn/tool calling**: Read [references/examples_reference.md](references/examples_reference.md) Search-R1 section
- **Custom reward models**: See custom RM pattern in examples reference
- **FSDP instead of Megatron**: Use `--train-backend fsdp`, skip weight conversion
- **Large-scale training**: See multi-node examples (GLM-4.5, DeepSeek-R1)
- **Source code exploration**: Check [references/source_code_reference.md](references/source_code_reference.md)

## Documentation Navigation

SLIME has extensive documentation. Use this guide to find what you need quickly.

### Essential Documentation (Read These First)
1. **Quick Start Guide**: `docs/en/get_started/quick_start.md` - Setup and first training run
2. **Usage Guide**: `docs/en/get_started/usage.md` - Comprehensive parameter reference
3. **Example Docs**: `docs/en/examples/qwen3-4B.md` or `docs/en/examples/glm4-9B.md`

For detailed navigation of all documentation, see [references/doc_navigation.md](references/doc_navigation.md).

### Common Tasks → Documentation Mapping

| Task | Documentation |
|------|---------------|
| First-time setup | `docs/en/get_started/quick_start.md` |
| Understanding parameters | `docs/en/get_started/usage.md` |
| Basic training (8 GPUs) | `docs/en/examples/qwen3-4B.md` |
| Multi-turn tool use | `examples/search-r1/` |
| Custom generation logic | `docs/en/get_started/customization.md` |
| Multi-node training | `docs/en/examples/glm4.5-355B-A32B.md` |
| FSDP backend | `docs/en/get_started/usage.md` (FSDP section) |
| VLM training | `examples/geo3k_vlm/` |
| Troubleshooting | `docs/en/get_started/qa.md` |

## Core Concepts

### Training Loop
SLIME uses a "Rollout → Train" loop:
1. **Rollout**: Generate responses using SGLang inference
2. **Reward**: Compute rewards using reward model
3. **Train**: Update model weights using Megatron/FSDP
4. Repeat for `--num-rollout` iterations

### Key Constraint
```
rollout-batch-size × n-samples-per-prompt = global-batch-size × num-steps-per-rollout
```

### Resource Allocation Modes

**Colocated** (training and inference share GPUs):
```bash
--actor-num-nodes 1 \
--actor-num-gpus-per-node 8 \
--colocate \
--sglang-mem-fraction-static 0.7
```

**Disaggregated** (separate GPUs for training/inference):
```bash
--actor-num-nodes 1 \
--actor-num-gpus-per-node 4 \
--rollout-num-gpus 4
```

## Parameter Quick Reference

### Essential Parameters

**Model Loading**:
- `--hf-checkpoint`: HuggingFace model path (for SGLang and FSDP)
- `--ref-load`: Megatron reference model checkpoint
- `--load`: Megatron actor checkpoint (resume training)
- `--save`: Save path for checkpoints

**Data**:
- `--prompt-data`: JSONL dataset path
- `--input-key`: Field name for prompts (default: "prompt")
- `--label-key`: Field name for labels (default: "label")
- `--metadata-key`: Field name for metadata (default: "metadata")
- `--apply-chat-template`: Apply tokenizer chat template

**Rollout**:
- `--rollout-batch-size`: Prompts per rollout
- `--n-samples-per-prompt`: Responses per prompt
- `--rollout-max-response-len`: Max response length
- `--rollout-temperature`: Sampling temperature

**Training**:
- `--num-rollout`: Total training iterations
- `--num-steps-per-rollout`: Optimizer steps per rollout (default: 1)
- `--global-batch-size`: Samples per optimizer step
- `--advantage-estimator`: RL algorithm (grpo, gspo, ppo, reinforce_plus_plus)

**Reward Model**:
- `--rm-type`: Built-in RM type (e.g., "deepscaler")
- `--custom-rm-path`: Custom RM function path

**Backends**:
- `--train-backend`: Training backend (megatron or fsdp)
- `--rollout-num-gpus-per-engine`: GPUs per SGLang engine (like tp_size)

For complete parameter reference, see `docs/en/get_started/usage.md`.

## Common Workflows

### 1. Standard Single-Turn Training

Use example scripts as templates:
- `scripts/run-qwen3-4B.sh`: Basic 8xH100 setup
- `scripts/run-glm4-9B.sh`: With dynamic sampling

Key sections in script:
```bash
# Load model config
source scripts/models/qwen3-4B.sh

# Configure checkpoints
CKPT_ARGS=(--hf-checkpoint /root/Qwen3-4B ...)

# Configure rollout
ROLLOUT_ARGS=(
  --rollout-batch-size 32
  --n-samples-per-prompt 8
  --rm-type deepscaler
)

# Configure algorithm
GRPO_ARGS=(--advantage-estimator grpo ...)

# Run training
ray job submit ... -- python3 train.py \
  ${MODEL_ARGS[@]} ${CKPT_ARGS[@]} ${ROLLOUT_ARGS[@]} ...
```

### 2. Multi-Turn Tool Calling

For multi-turn scenarios (like Search-R1):

1. **Prepare Data** with metadata:
   ```json
   {
     "question": "User query",
     "final_answer": "Expected answer",
     "metadata": "{\"session_id\": \"123\", \"tool_code\": \"...\"}"
   }
   ```

2. **Implement Custom Generation Function**:
   ```python
   async def generate(args, sample: Sample, sampling_params) -> Sample:
       for turn in range(max_turns):
           # Generate action
           model_output = await call_sglang(...)
           sample.loss_mask += [1] * len(model_tokens)  # Train on actions

           # Execute tool
           tool_output = await execute_tool(...)
           sample.loss_mask += [0] * len(tool_tokens)  # Mask tool outputs

           if action == "answer":
               break

       sample.tokens = prompt_tokens + response_tokens
       sample.response_length = len(response_tokens)
       return sample
   ```

3. **Configure Custom Functions**:
   ```bash
   --custom-generate-function-path my_module.generate \
   --custom-rm-path my_module.reward_func \
   --metadata-key metadata
   ```

See `examples/search-r1/` for complete example.

### 3. Dynamic Sampling (DAPO-style)

Filter low-quality samples during generation:

```bash
ROLLOUT_ARGS+=(
  --over-sampling-batch-size 64 \
  --rollout-batch-size 32 \
  --dynamic-sampling-filter-path \
    slime.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std
)
```

How it works:
- Samples 64 prompts (over-sampling)
- Filters groups based on reward diversity
- Keeps only 32 prompts × 8 samples that pass filter
- Automatically resamples if too many filtered out

### 4. FSDP Backend (No Weight Conversion)

```bash
--train-backend fsdp \
--hf-checkpoint /root/Qwen3-4B \
--gradient-checkpointing \
--context-parallel-size 2
```

Benefits:
- No HF → Megatron weight conversion needed
- Directly load HuggingFace checkpoints
- Simpler setup for supported models

See `examples/geo3k_vlm/` and `docs/en/get_started/usage.md` FSDP section.

### 5. Multi-Node Training

1. Start Ray cluster:
   ```bash
   # Head node
   ray start --head --node-ip-address ${MASTER_ADDR} --num-gpus 8

   # Worker nodes
   ray start --address=${MASTER_ADDR}:6379 --num-gpus 8
   ```

2. Submit job:
   ```bash
   ray job submit --address="http://127.0.0.1:8265" \
     --runtime-env-json='{"env_vars": {"PYTHONPATH": "/root/Megatron-LM/"}}' \
     -- python3 train.py \
     --actor