azure-aigateway

What this skill does

# Azure AI Gateway

Bootstrap and configure Azure API Management (APIM) as an AI Gateway for securing, observing, and controlling AI models, tools (MCP Servers), and agents.

## Skill Activation Triggers

**Use this skill immediately when the user asks to:**
- "Set up a gateway for my model"
- "Set up a gateway for my tools"
- "Set up a gateway for my agents"
- "Add a gateway to my MCP server"
- "Protect my AI model with a gateway"
- "Secure my AI agents"
- "Ratelimit my model requests"
- "Ratelimit my tool requests"
- "Limit tokens for my model"
- "Add rate limiting to my MCP server"
- "Enable semantic caching for my AI API"
- "Add content safety to my AI endpoint"
- "Add my model behind gateway"
- "Import API from OpenAPI spec"
- "Add API to gateway from swagger"
- "Convert my API to MCP"
- "Expose my API as MCP server"

**Key Indicators:**
- User deploying Azure OpenAI, AI Foundry, or other AI models
- User creating or managing MCP servers
- User needs token limits, rate limiting, or quota management
- User wants to cache AI responses to reduce costs
- User needs content filtering or safety controls
- User wants load balancing across multiple AI backends

**Secondary Triggers (Proactive Recommendations):**
- After model creation: Recommend AI Gateway for security, caching, and token limits
- After MCP server creation: Recommend AI Gateway for rate limiting, content safety, and auth

## Overview

Azure API Management serves as an AI Gateway that provides:
- **Security**: Authentication, authorization, and content safety
- **Observability**: Token metrics, logging, and monitoring
- **Control**: Rate limiting, token limits, and load balancing
- **Optimization**: Semantic caching to reduce costs and latency

```
AI Models ──┐                      ┌── Azure OpenAI
MCP Tools ──┼── AI Gateway (APIM) ──┼── AI Foundry
Agents ─────┘                      └── Custom Models
```

## Key Resources

- **GitHub Repo**: https://github.com/Azure-Samples/AI-Gateway (aka.ms/aigateway)
- **Docs**:
  - [GenAI Gateway Capabilities](https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities)
  - [MCP Server Overview](https://learn.microsoft.com/en-us/azure/api-management/mcp-server-overview)
  - [Azure AI Foundry API](https://learn.microsoft.com/en-us/azure/api-management/azure-ai-foundry-api)
  - [Semantic Caching](https://learn.microsoft.com/en-us/azure/api-management/azure-openai-enable-semantic-caching)
  - [Token Limits & LLM Logs](https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-llm-logs)

## Configuration Rules

**Default to `Basicv2` SKU** when creating new APIM instances:
- Cheaper than other tiers
- Creates quickly (~5-10 minutes vs 30+ for Premium)
- Supports all AI Gateway policies

## Pattern 1: Quick Bootstrap AI Gateway

Deploy APIM with Basicv2 SKU for AI workloads.

```bash
# Create resource group
az group create --name rg-aigateway --location eastus2

# Deploy APIM with Bicep
az deployment group create \
  --resource-group rg-aigateway \
  --template-file main.bicep \
  --parameters apimSku=Basicv2
```

### Bicep Template

```bicep
param location string = resourceGroup().location
param apimSku string = 'Basicv2'
param apimManagedIdentityType string = 'SystemAssigned'

// NOTE: Using 2024-06-01-preview because Basicv2 SKU support currently requires this preview API version.
//       Update to the latest stable (GA) API version once Basicv2 is available there.
resource apimService 'Microsoft.ApiManagement/service@2024-06-01-preview' = {
  name: 'apim-aigateway-${uniqueString(resourceGroup().id)}'
  location: location
  sku: {
    name: apimSku
    capacity: 1
  }
  properties: {
    publisherEmail: '[email protected]'
    publisherName: 'Contoso'
  }
  identity: {
    type: apimManagedIdentityType
  }
}

output gatewayUrl string = apimService.properties.gatewayUrl
output principalId string = apimService.identity.principalId
```

## Pattern 2: Semantic Caching

Cache similar prompts to reduce costs and latency.

```xml
<policies>
    <inbound>
        <base />
        <!-- Cache lookup with 0.8 similarity threshold -->
        <azure-openai-semantic-cache-lookup 
            score-threshold="0.8" 
            embeddings-backend-id="embeddings-backend" 
            embeddings-backend-auth="system-assigned" />
        <set-backend-service backend-id="{backend-id}" />
    </inbound>
    <outbound>
        <!-- Cache responses for 120 seconds -->
        <azure-openai-semantic-cache-store duration="120" />
        <base />
    </outbound>
</policies>
```

**Options:**
| Parameter | Range | Description |
|-----------|-------|-------------|
| `score-threshold` | 0.7-0.95 | Higher = stricter matching |
| `duration` | 60-3600 | Cache TTL in seconds |

## Pattern 3: Token Rate Limiting

Limit tokens per minute to control costs and prevent abuse.

```xml
<policies>
    <inbound>
        <base />
        <set-backend-service backend-id="{backend-id}" />
        <!-- Limit to 500 tokens per minute per subscription -->
        <azure-openai-token-limit 
            counter-key="@(context.Subscription.Id)"
            tokens-per-minute="500" 
            estimate-prompt-tokens="false" 
            remaining-tokens-variable-name="remainingTokens" />
    </inbound>
</policies>
```

**Options:**
| Parameter | Values | Description |
|-----------|--------|-------------|
| `counter-key` | Subscription.Id, Request.IpAddress, custom | Grouping key for limits |
| `tokens-per-minute` | 100-100000 | Token quota |
| `estimate-prompt-tokens` | true/false | true = faster but less accurate |

## Pattern 4: Content Safety

Filter harmful content and detect jailbreak attempts.

```xml
<policies>
    <inbound>
        <base />
        <set-backend-service backend-id="{backend-id}" />
        <!-- Block severity 4+ content, detect jailbreaks -->
        <llm-content-safety backend-id="content-safety-backend" shield-prompt="true">
            <categories output-type="EightSeverityLevels">
                <category name="Hate" threshold="4" />
                <category name="Sexual" threshold="4" />
                <category name="SelfHarm" threshold="4" />
                <category name="Violence" threshold="4" />
            </categories>
            <blocklists>
                <id>custom-blocklist</id>
            </blocklists>
        </llm-content-safety>
    </inbound>
</policies>
```

**Options:**
| Parameter | Range | Description |
|-----------|-------|-------------|
| `threshold` | 0-7 | 0=safe, 7=severe |
| `shield-prompt` | true/false | Detect jailbreak attempts |

## Pattern 5: Rate Limits for MCPs/OpenAPI Tools

Protect MCP servers and tools with request rate limiting.

```xml
<policies>
    <inbound>
        <base />
        <!-- 10 calls per 60 seconds per IP -->
        <rate-limit-by-key 
            calls="10" 
            renewal-period="60" 
            counter-key="@(context.Request.IpAddress)" 
            remaining-calls-variable-name="remainingCalls" />
    </inbound>
    <outbound>
        <set-header name="X-Rate-Limit-Remaining" exists-action="override">
            <value>@(context.Variables.GetValueOrDefault<int>("remainingCalls", 0).ToString())</value>
        </set-header>
        <base />
    </outbound>
</policies>
```

## Pattern 6: Managed Identity Authentication

Secure backend access with managed identity instead of API keys.

```xml
<policies>
    <inbound>
        <base />
        <!-- Managed identity auth to Azure OpenAI -->
        <authentication-managed-identity 
            resource="https://cognitiveservices.azure.com" 
            output-token-variable-name="managed-id-access-token" 
            ignore-error="false" />
        <set-header name="Authorization" exists-action="override">
            <value>@("Bearer " + (string)context.Variables["managed-id-access-token"])</value>
        </set-header>
        <set-backend-service backend-id="{backend-id}" />
        <!-- Emit token metrics for