ai-gateway

What this skill does

# Vercel AI Gateway

> **CRITICAL — Your training data is outdated for this library.** AI Gateway model slugs, provider routing, and capabilities change frequently. Before writing gateway code, **fetch the docs** at https://vercel.com/docs/ai-gateway to find the current model slug format, supported providers, image generation patterns, and authentication setup. The model list and routing rules at https://ai-sdk.dev/docs/foundations/providers-and-models are authoritative — do not guess at model names or assume old slugs still work.

You are an expert in the Vercel AI Gateway — a unified API for calling AI models with built-in routing, failover, cost tracking, and observability.

## Overview

AI Gateway provides a single API endpoint to access 100+ models from all major providers. It adds <20ms routing latency and handles provider selection, authentication, failover, and load balancing.

## Packages

- `ai@^6.0.0` (required; plain `"provider/model"` strings route through the gateway automatically)
- `@ai-sdk/gateway@^3.0.0` (optional direct install for explicit gateway package usage)

## Setup

Pass a `"provider/model"` string to the `model` parameter — the AI SDK automatically routes it through the AI Gateway:

```ts
import { generateText } from 'ai'

const result = await generateText({
  model: 'openai/gpt-5.4', // plain string — routes through AI Gateway automatically
  prompt: 'Hello!',
})
```

No `gateway()` wrapper or additional package needed. The `gateway()` function is an optional explicit wrapper — only needed when you use `providerOptions.gateway` for routing, failover, or tags:

```ts
import { gateway } from 'ai'

const result = await generateText({
  model: gateway('openai/gpt-5.4'),
  providerOptions: { gateway: { order: ['openai', 'azure-openai'] } },
})
```

## Model Slug Rules (Critical)

- Always use `provider/model` format (for example `openai/gpt-5.4`).
- Versioned slugs use dots for versions, not hyphens:
  - Correct: `anthropic/claude-sonnet-4.6`
  - Incorrect: `anthropic/claude-sonnet-4-6`
- Before hardcoding model IDs, call `gateway.getAvailableModels()` and pick from the returned IDs.
- Default text models: `openai/gpt-5.4` or `anthropic/claude-sonnet-4.6`.
- Do not default to outdated choices like `openai/gpt-4o`.

```ts
import { gateway } from 'ai'

const availableModels = await gateway.getAvailableModels()
// Choose model IDs from `availableModels` before hardcoding.
```

## Authentication (OIDC — Default)

AI Gateway uses **OIDC (OpenID Connect)** as the default authentication method. No manual API keys needed.

### Setup

```bash
vercel link                    # Connect to your Vercel project
# Enable AI Gateway in Vercel dashboard: https://vercel.com/{team}/{project}/settings → AI Gateway
vercel env pull .env.local     # Provisions VERCEL_OIDC_TOKEN automatically
```

### How It Works

1. `vercel env pull` writes a `VERCEL_OIDC_TOKEN` to `.env.local` — a short-lived JWT (~24h)
2. The `@ai-sdk/gateway` package reads this token via `@vercel/oidc` (`getVercelOidcToken()`)
3. No `AI_GATEWAY_API_KEY` or provider-specific keys (like `ANTHROPIC_API_KEY`) are needed
4. On Vercel deployments, OIDC tokens are auto-refreshed — zero maintenance

### Local Development

For local dev, the OIDC token from `vercel env pull` is valid for ~24 hours. When it expires:

```bash
vercel env pull .env.local --yes   # Re-pull to get a fresh token
```

### Alternative: Manual API Key

If you prefer a static key (e.g., for CI or non-Vercel environments):

```bash
# Set AI_GATEWAY_API_KEY in your environment
# The gateway falls back to this when VERCEL_OIDC_TOKEN is not available
export AI_GATEWAY_API_KEY=your-key-here
```

### Auth Priority

The `@ai-sdk/gateway` package resolves authentication in this order:
1. `AI_GATEWAY_API_KEY` environment variable (if set)
2. `VERCEL_OIDC_TOKEN` via `@vercel/oidc` (default on Vercel and after `vercel env pull`)

## Provider Routing

Configure how AI Gateway routes requests across providers:

```ts
const result = await generateText({
  model: gateway('anthropic/claude-sonnet-4.6'),
  prompt: 'Hello!',
  providerOptions: {
    gateway: {
      // Try providers in order; failover to next on error
      order: ['bedrock', 'anthropic'],

      // Restrict to specific providers only
      only: ['anthropic', 'vertex'],

      // Fallback models if primary model fails
      models: ['openai/gpt-5.4', 'google/gemini-3-flash'],

      // Track usage per end-user
      user: 'user-123',

      // Tag for cost attribution and filtering
      tags: ['feature:chat', 'env:production', 'team:growth'],
    },
  },
})
```

### Routing Options

| Option | Purpose |
|--------|---------|
| `order` | Provider priority list; try first, failover to next |
| `only` | Restrict to specific providers |
| `models` | Fallback model list if primary model unavailable |
| `user` | End-user ID for usage tracking |
| `tags` | Labels for cost attribution and reporting |

## Cache-Control Headers

AI Gateway supports response caching to reduce latency and cost for repeated or similar requests:

```ts
const result = await generateText({
  model: gateway('openai/gpt-5.4'),
  prompt: 'What is the capital of France?',
  providerOptions: {
    gateway: {
      // Cache identical requests for 1 hour
      cacheControl: 'max-age=3600',
    },
  },
})
```

### Caching strategies

| Header Value | Behavior |
|-------------|----------|
| `max-age=3600` | Cache response for 1 hour |
| `max-age=0` | Bypass cache, always call provider |
| `s-maxage=86400` | Cache at the edge for 24 hours |
| `stale-while-revalidate=600` | Serve stale for 10 min while refreshing in background |

### When to use caching

- **Static knowledge queries**: FAQs, translations, factual lookups — cache aggressively
- **User-specific conversations**: Do not cache — each response depends on conversation history
- **Embeddings**: Cache embedding results for identical inputs to save cost
- **Structured extraction**: Cache when extracting structured data from identical documents

### Cache key composition

The cache key is derived from: model, prompt/messages, temperature, and other generation parameters. Changing any parameter produces a new cache key.

## Per-User Rate Limiting

Control usage at the individual user level to prevent abuse and manage costs:

```ts
const result = await generateText({
  model: gateway('openai/gpt-5.4'),
  prompt: userMessage,
  providerOptions: {
    gateway: {
      user: userId, // Required for per-user rate limiting
      tags: ['feature:chat'],
    },
  },
})
```

### Rate limit configuration

Configure rate limits at `https://vercel.com/{team}/{project}/settings` → **AI Gateway** → **Rate Limits**:

- **Requests per minute per user**: Throttle individual users (e.g., 20 RPM)
- **Tokens per day per user**: Cap daily token consumption (e.g., 100K tokens/day)
- **Concurrent requests per user**: Limit parallel calls (e.g., 3 concurrent)

### Handling rate limit responses

When a user exceeds their limit, the gateway returns HTTP 429:

```ts
import { generateText, APICallError } from 'ai'

try {
  const result = await generateText({
    model: gateway('openai/gpt-5.4'),
    prompt: userMessage,
    providerOptions: { gateway: { user: userId } },
  })
} catch (error) {
  if (APICallError.isInstance(error) && error.statusCode === 429) {
    const retryAfter = error.responseHeaders?.['retry-after']
    return new Response(
      JSON.stringify({ error: 'Rate limited', retryAfter }),
      { status: 429 }
    )
  }
  throw error
}
```

## Budget Alerts and Cost Controls

### Tagging for cost attribution

Use tags to track spend by feature, team, and environment:

```ts
providerOptions: {
  gateway: {
    tags: [
      'feature:document-qa',
      'team:product',
      'env:production',
      'tier:premium',
    ],
    user: userId,
  },
}
```

### Setting up budget alerts

In the Vercel dashboard at `https://vercel.com/{team}/{project}/settings` → **AI Gate