model-registry-maintainer

What this skill does

# Model Registry Maintainer

This skill provides guidance for maintaining MassGen's model registry across two key files:

1. **`massgen/backend/capabilities.py`** - Models, capabilities, release dates
2. **`massgen/token_manager/token_manager.py`** - Pricing, context windows

## When to Use This Skill

- New model released by a provider
- Model pricing changes
- Context window limits updated
- Model capabilities changed
- New provider/backend added

## Two Files to Maintain

### File 1: capabilities.py (Models & Features)

**What it contains:**
- List of available models per provider
- Model capabilities (web search, code execution, vision, etc.)
- Release dates
- Default models

**Used by:**
- Config builder (`--quickstart`, `--generate-config`)
- Documentation generation
- Backend validation

**Always update this file** for new models.

### File 2: token_manager.py (Pricing & Limits)

**What it contains:**
- Hardcoded pricing/context windows for models NOT in LiteLLM database
- On-demand loading from LiteLLM database (500+ models)

**Used by:**
- Cost estimation
- Token counting
- Context management

**Pricing resolution order:**
1. LiteLLM database (fetched on-demand, cached 1 hour)
2. Hardcoded PROVIDER_PRICING (fallback only)
3. Pattern matching heuristics

**Only update PROVIDER_PRICING if:**
- Model is NOT in LiteLLM database
- LiteLLM pricing is incorrect/outdated
- Model is custom/internal to your organization

## Information to Gather for New Models

### 1. Release Date
- Format: `"YYYY-MM"`
- Sources:
  - OpenAI: https://openai.com/index
  - Anthropic: https://www.anthropic.com/news
  - Google DeepMind: https://blog.google/technology/google-deepmind/
  - xAI: https://x.ai/news

### 2. Context Window
- Input context size (tokens)
- Max output tokens
- Look for: "context window", "max tokens", "input/output limits"

### 3. Pricing
- Input cost per 1K tokens (USD)
- Output cost per 1K tokens (USD)
- Cached input cost (if applicable)
- Sources:
  - OpenAI: https://openai.com/api/pricing/
  - Anthropic: https://www.anthropic.com/pricing
  - Google: https://ai.google.dev/pricing
  - xAI: https://x.ai/api/pricing

### 4. Capabilities
- Web search, code execution, vision, reasoning, etc.
- Check official API documentation

### 5. Model Name
- Exact API identifier (case-sensitive)
- Check provider's model documentation

## Adding a New Model - Complete Workflow

### Step 1: Add to capabilities.py

Add model to the `models` list and `model_release_dates`:

```python
# massgen/backend/capabilities.py

"openai": BackendCapabilities(
    # ... existing fields ...
    models=[
        "new-model-name",  # Add here (newest first)
        "gpt-5.1",
        # ... existing models ...
    ],
    model_release_dates={
        "new-model-name": "2025-12",  # Add here
        "gpt-5.1": "2025-11",
        # ... existing dates ...
    },
)
```

### Step 2: Check if pricing is in LiteLLM (Usually Skip)

**First, check if the model is already in LiteLLM database:**

```python
import requests

url = "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"
pricing_db = requests.get(url).json()

if "new-model-name" in pricing_db:
    print("✅ Model found in LiteLLM - no need to update token_manager.py")
    print(f"Pricing: ${pricing_db['new-model-name']['input_cost_per_token']*1000}/1K input")
else:
    print("❌ Model NOT in LiteLLM - need to add to PROVIDER_PRICING")
```

**Only if NOT in LiteLLM**, add to `PROVIDER_PRICING`:

```python
# massgen/token_manager/token_manager.py

PROVIDER_PRICING: Dict[str, Dict[str, ModelPricing]] = {
    "OpenAI": {
        # Format: ModelPricing(input_per_1k, output_per_1k, context_window, max_output)
        "new-model-name": ModelPricing(0.00125, 0.01, 300000, 150000),
        # ... existing models ...
    },
}
```

**Provider name mapping**:
- `"OpenAI"` (not "openai")
- `"Anthropic"` (not "claude")
- `"Google"` (not "gemini")
- `"xAI"` (not "grok")

### Step 3: Update Capabilities (if new features)

If the model introduces new capabilities:

```python
supported_capabilities={
    "web_search",
    "code_execution",
    "new_capability",  # Add here
}
```

### Step 4: Update Default Model (if appropriate)

Only change if the new model should be the recommended default:

```python
default_model="new-model-name"
```

### Step 5: Validate and Test

```bash
# Run capabilities tests
uv run pytest massgen/tests/test_backend_capabilities.py -v

# Test config generation with new model
massgen --generate-config ./test.yaml --config-backend openai --config-model new-model-name

# Verify the config was created successfully
cat ./test.yaml
```

### Step 6: Regenerate Documentation

```bash
uv run python docs/scripts/generate_backend_tables.py
cd docs && make html
```

## Current Model Data

### OpenAI Models (as of Nov 2025)

In capabilities.py:
```python
models=[
    "gpt-5.1",        # 2025-11
    "gpt-5-codex",    # 2025-09
    "gpt-5",          # 2025-08
    "gpt-5-mini",     # 2025-08
    "gpt-5-nano",     # 2025-08
    "gpt-4.1",        # 2025-04
    "gpt-4.1-mini",   # 2025-04
    "gpt-4.1-nano",   # 2025-04
    "gpt-4o",         # 2024-05
    "gpt-4o-mini",    # 2024-07
    "o4-mini",        # 2025-04
]
```

In token_manager.py (add missing models):
```python
"OpenAI": {
    "gpt-5": ModelPricing(0.00125, 0.01, 400000, 128000),
    "gpt-5-mini": ModelPricing(0.00025, 0.002, 400000, 128000),
    "gpt-5-nano": ModelPricing(0.00005, 0.0004, 400000, 128000),
    "gpt-4o": ModelPricing(0.0025, 0.01, 128000, 16384),
    "gpt-4o-mini": ModelPricing(0.00015, 0.0006, 128000, 16384),
    # Missing: gpt-5.1, gpt-5-codex, gpt-4.1 family, o4-mini
}
```

### Claude Models (as of Nov 2025)

In capabilities.py:
```python
models=[
    "claude-haiku-4-5-20251001",    # 2025-10
    "claude-sonnet-4-5-20250929",   # 2025-09
    "claude-opus-4-1-20250805",     # 2025-08
    "claude-sonnet-4-20250514",     # 2025-05
]
```

In token_manager.py:
```python
"Anthropic": {
    "claude-haiku-4-5": ModelPricing(0.001, 0.005, 200000, 65536),
    "claude-sonnet-4-5": ModelPricing(0.003, 0.015, 200000, 65536),
    "claude-opus-4.1": ModelPricing(0.015, 0.075, 200000, 32768),
    "claude-sonnet-4": ModelPricing(0.003, 0.015, 200000, 8192),
}
```

### Gemini Models (as of Nov 2025)

In capabilities.py:
```python
models=[
    "gemini-3-pro-preview",  # 2025-11
    "gemini-2.5-flash",      # 2025-06
    "gemini-2.5-pro",        # 2025-06
]
```

In token_manager.py (missing gemini-2.5 and gemini-3):
```python
"Google": {
    "gemini-1.5-pro": ModelPricing(0.00125, 0.005, 2097152, 8192),
    "gemini-1.5-flash": ModelPricing(0.000075, 0.0003, 1048576, 8192),
    # Missing: gemini-2.5-pro, gemini-2.5-flash, gemini-3-pro-preview
}
```

### Grok Models (as of Nov 2025)

In capabilities.py:
```python
models=[
    "grok-4-1-fast-reasoning",      # 2025-11
    "grok-4-1-fast-non-reasoning",  # 2025-11
    "grok-code-fast-1",             # 2025-08
    "grok-4",                       # 2025-07
    "grok-4-fast",                  # 2025-09
    "grok-3",                       # 2025-02
    "grok-3-mini",                  # 2025-05
]
```

In token_manager.py (missing grok-3, grok-4 families):
```python
"xAI": {
    "grok-2-latest": ModelPricing(0.005, 0.015, 131072, 131072),
    "grok-2": ModelPricing(0.005, 0.015, 131072, 131072),
    "grok-2-mini": ModelPricing(0.001, 0.003, 131072, 65536),
    # Missing: grok-3, grok-4, grok-4-1 families
}
```

## Model Name Matching

**Important:** The names in `PROVIDER_PRICING` use simplified patterns:

- `"gpt-5"` matches `gpt-5`, `gpt-5-preview`, `gpt-5-*`
- `"claude-sonnet-4-5"` matches `claude-sonnet-4-5-*` (any date suffix)
- `"gemini-2.5-pro"` is exact match

The token manager uses prefix matching for flexibility.

## Common Tasks

### Task: Add brand new GPT-5.2 model

1. Research: Release date, pricing, context window, capabilities
2. Add to `capabilities.py` models list and release_dates
3. A