identityserver-deployment

What this skill does

# IdentityServer Deployment, Proxies, and Production Readiness

## When to Use This Skill

- Deploying IdentityServer behind a reverse proxy or load balancer
- Configuring ASP.NET Core Data Protection for production persistence
- Implementing health checks for monitoring IdentityServer instances
- Setting up distributed caching for multi-instance deployments
- Configuring OpenTelemetry for metrics, traces, and logs
- Troubleshooting common deployment issues (HTTPS downgrade, cookie problems, key rotation failures)
- Understanding the difference between Data Protection keys and IdentityServer signing keys
- Setting up logging and events for production monitoring

Docs: https://docs.duendesoftware.com/identityserver/deployment

## Deployment Architecture

IdentityServer is ASP.NET Core middleware. It can be hosted with the same diversity of technology as any ASP.NET Core application:

- **Hosting**: On-premises, cloud (Azure, AWS, GCP), containers, Kubernetes
- **Web servers**: Kestrel, IIS, Nginx, Apache
- **Artifacts**: Files, containers (no Dockerfile needed with `dotnet publish /t:PublishContainer`)
- **Scaling**: Horizontal with load balancers; requires shared state for multi-instance

## Reverse Proxy and Load Balancer Configuration

### The Problem

When IdentityServer runs behind a proxy that terminates TLS or changes the originating IP, the middleware sees incorrect request information. This causes:

- HTTPS requests downgraded to HTTP
- HTTP issuer published in `.well-known/openid-configuration` instead of HTTPS
- Incorrect host names in discovery document or redirects
- Cookies missing the `Secure` attribute (breaks `SameSite` behavior)

### Solution: ForwardedHeaders Middleware

Most proxies set `X-Forwarded-For` and `X-Forwarded-Proto` headers. Configure ASP.NET Core to read them.

#### Option 1: Environment Variable (Simplest)

Set `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true`. This automatically adds the middleware and accepts forwarded headers from any single proxy. Best for cloud-hosted environments and Kubernetes.

#### Option 2: Explicit Configuration (More Control)

```csharp
// Program.cs
builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders = ForwardedHeaders.XForwardedHost |
                                ForwardedHeaders.XForwardedProto;

    // Add the IP address of your known proxy
    options.KnownProxies.Add(IPAddress.Parse("203.0.113.42"));

    // Or use a network range
    // var network = new IPNetwork(IPAddress.Parse("198.51.100.0"), 24);
    // options.KnownNetworks.Add(network);

    // Number of proxies in front of the app
    options.ForwardLimit = 1;
});
```

**Important**: The ForwardedHeaders middleware must run **early** in the pipeline, before IdentityServer middleware and ASP.NET authentication middleware.

### Default KnownNetworks

By default, `KnownNetworks` and `KnownProxies` support localhost (`127.0.0.1/8` and `::1`). This is useful for local development or when the proxy and .NET host are on the same machine. In production, configure the actual proxy addresses.

## ASP.NET Core Data Protection

> **Cross-cutting concern:** Data protection is critical for all Duende products — both IdentityServer and BFF. See [ASP.NET Core Data Protection](https://docs.duendesoftware.com/general/data-protection/) for comprehensive guidance covering all Duende SDKs.

### Why It Matters

Data Protection is critical for IdentityServer. It encrypts and signs sensitive data including:

- Signing keys at rest (when automatic key management is used)
- Persisted grants at rest
- Server-side session data at rest
- State parameters for external OIDC providers
- UI message payloads (logout context, error context)
- Authentication session cookies
- Anti-forgery tokens

### Production Configuration

```csharp
// Program.cs
builder.Services.AddDataProtection()
    // Choose a persistence method
    .PersistKeysToFoo()       // PersistKeysToFileSystem, PersistKeysToDbContext,
                               // PersistKeysToAzureBlobStorage, PersistKeysToAWSSystemsManager,
                               // PersistKeysToStackExchangeRedis
    // Choose a key protection method
    .ProtectKeysWithBar()     // ProtectKeysWithCertificate, ProtectKeysWithAzureKeyVault
    // Set explicit application name
    .SetApplicationName("My.IdentityServer");
```

### Critical Rules

1. **Always persist keys to durable storage** using a `.PersistKeysTo...()` method
2. **Ensure the storage itself is durable** — e.g., if using Redis, configure Redis persistence (RDB/AOF)
3. **Always set an explicit application name** with `.SetApplicationName()` to prevent key isolation issues
4. **Share keys across all load-balanced instances**
5. **Consider a key escrow sink** — for backup/restore of corrupted data protection keys, configure an `IXmlEncryptor`-based escrow

### Data Protection Keys vs Signing Keys

| Aspect       | Data Protection Keys                               | IdentityServer Signing Keys                                               |
| ------------ | -------------------------------------------------- | ------------------------------------------------------------------------- |
| Purpose      | Encrypt/sign sensitive data at rest and in cookies | Sign JWT tokens (id_tokens, access tokens)                                |
| Cryptography | Symmetric (private key)                            | Asymmetric (public/private key pair)                                      |
| Visibility   | Internal to the application                        | Public keys published via discovery/JWKS                                  |
| Managed by   | ASP.NET Core framework                             | IdentityServer (automatic key management)                                 |
| Storage      | Configured via `.PersistKeysTo...()`               | File system (default), EF operational store, or custom `ISigningKeyStore` |

Both are critical secrets. Losing either causes failures.

### Common Data Protection Problems

| Problem                                     | Symptom                                                 | Solution                                                  |
| ------------------------------------------- | ------------------------------------------------------- | --------------------------------------------------------- |
| No shared keys in load-balanced environment | `CryptographicException`: key not found in key ring     | Configure shared key persistence                          |
| Keys generated in dev included in build     | Keys from wrong environment can't be read in production | Exclude `~/keys` directory from source control and builds |
| Application name mismatch                   | Keys from one deployment can't be read by another       | Set explicit `SetApplicationName()` consistently          |
| IIS lacking permissions                     | Ephemeral keys generated every restart                  | Follow Microsoft's IIS Data Protection configuration      |
| .NET 6 path normalization change            | Keys break between .NET versions                        | Always set explicit application name (reverted in .NET 7+) |

### Symptoms of Data Protection Failure

- `CryptographicException` in logs
- Error messages like "Error unprotecting key with kid {Signing Key ID}"
- "The key {Data Protection Key ID} was not found in the key ring"
- Automatic signing key management fails silently

## IdentityServer Data Stores for Multi-Instance

### Configuration Data

For multi-instance deployments, configuration data must be shared:

| Scenario                      | Recommendation                                                        |
| ----------------------------- | --------------------------------------------------------------------- |
| Rarely changing configuration | In-memory stores loaded from config files (with redeploy for changes) |
| Dynamic configuration (SaaS)  | Database via EF Core stores o