loki-config-generator

What this skill does

# Loki Configuration Generator

## Overview

Generate production-ready Grafana Loki server configurations with best practices. Supports monolithic, simple scalable, and microservices deployment modes with S3, GCS, Azure, or filesystem storage.

> **Current Stable:** Loki 3.6.2 (November 2025)
> **Important:** Promtail deprecated in 3.4 - use [Grafana Alloy](https://grafana.com/docs/alloy/latest/) instead. See `examples/grafana-alloy.alloy` for the Alloy pipeline and `examples/grafana-alloy-daemonset.yaml` for the Kubernetes deployment.

## When to Use

Invoke when: deploying Loki, creating configs from scratch, migrating to Loki, implementing multi-tenant logging, configuring storage backends, or optimizing existing deployments.

---

## Generation Methods

### Method 1: Script Generation (Recommended)

**Use `scripts/generate_config.py` for consistent, validated configurations:**

```bash
# Simple Scalable with S3 (production)
python3 scripts/generate_config.py \
  --mode simple-scalable \
  --storage s3 \
  --bucket my-loki-bucket \
  --region us-east-1 \
  --retention-days 30 \
  --otlp-enabled \
  --output loki-config.yaml

# Monolithic with filesystem (development)
python3 scripts/generate_config.py \
  --mode monolithic \
  --storage filesystem \
  --no-auth-enabled \
  --output loki-dev.yaml

# Production with Thanos storage (Loki 3.4+)
python3 scripts/generate_config.py \
  --mode simple-scalable \
  --storage s3 \
  --thanos-storage \
  --otlp-enabled \
  --time-sharding \
  --output loki-thanos.yaml
```

**Script Options:**
| Option | Description |
|--------|-------------|
| `--mode` | monolithic, simple-scalable, microservices |
| `--storage` | filesystem, s3, gcs, azure |
| `--auth-enabled` / `--no-auth-enabled` | Explicitly enable/disable auth |
| `--otlp-enabled` | Enable OTLP ingestion configuration |
| `--thanos-storage` | Use Thanos object storage client (3.4+, cloud backends) |
| `--time-sharding` | Enable out-of-order ingestion (simple-scalable) |
| `--ruler` | Enable alerting/recording rules (not monolithic) |
| `--horizontal-compactor` | main/worker mode (simple-scalable, 3.6+) |
| `--zone-awareness` | Enable multi-AZ placement safeguards |
| `--limits-dry-run` | Log limit rejections without enforcing |

### Method 2: Manual Configuration

Follow the staged workflow below when script generation doesn't meet specific requirements or when learning the configuration structure.

### Output Formats

For Kubernetes deployments, generate BOTH formats:
1. **Native Loki config** (`loki-config.yaml`) - For ConfigMap or direct use
2. **Helm values** (`values.yaml`) - For Helm chart deployments

See `examples/kubernetes-helm-values.yaml` for Helm format.

---

## Documentation Lookup

### When to Use Context7/Web Search

**REQUIRED - Use Context7 MCP for:**
- Configuring features from Loki 3.4+ (Thanos storage, time sharding)
- Configuring features from Loki 3.6+ (horizontal compactor, enforced labels)
- Bloom filter configuration (complex, experimental)
- Custom OTLP attribute mappings beyond standard patterns
- Troubleshooting configuration errors

**OPTIONAL - Skip documentation lookup for:**
- Standard deployment modes (monolithic, simple-scalable)
- Basic storage configuration (S3, GCS, Azure, filesystem)
- Default limits and component settings
- Configurations covered in `references/` directory

### Context7 MCP (preferred)

```
resolve-library-id: "grafana loki"
get-library-docs: /websites/grafana_loki, topic: [component]
```

**Example topics:** `storage_config`, `limits_config`, `otlp`, `compactor`, `ruler`, `bloom`

### Web Search Fallback

Use when Context7 unavailable: `"Grafana Loki 3.6 [component] configuration documentation site:grafana.com"`

---

## Configuration Workflow

### Stage 1: Gather Requirements

**Deployment Mode:**
| Mode | Scale | Use Case |
|------|-------|----------|
| Monolithic | <100GB/day | Testing, development |
| Simple Scalable | 100GB-1TB/day | Production |
| Microservices | >1TB/day | Large-scale, multi-tenant |

**Storage Backend:** S3, GCS, Azure Blob, Filesystem, MinIO

**Key Questions:** Expected log volume? Retention period? Multi-tenancy needed? High availability requirements? Kubernetes deployment?

Ask the user directly if required information is missing.

### Stage 2: Schema Configuration (CRITICAL)

For all new deployments (Loki 2.9+), use TSDB with v13 schema:

```yaml
schema_config:
  configs:
    - from: "2025-01-01"  # Use deployment date
      store: tsdb
      object_store: s3     # s3, gcs, azure, filesystem
      schema: v13
      index:
        prefix: loki_index_
        period: 24h
```

**Key:** Schema cannot change after deployment without migration.

### Stage 3: Storage Configuration

**S3:**
```yaml
common:
  storage:
    s3:
      s3: s3://us-east-1/loki-bucket
      s3forcepathstyle: false
```

**GCS:** `gcs: { bucket_name: loki-bucket }`
**Azure:** `azure: { container_name: loki-container, account_name: ${AZURE_ACCOUNT_NAME} }`
**Filesystem:** `filesystem: { chunks_directory: /loki/chunks, rules_directory: /loki/rules }`

### Stage 4: Component Configuration

**Ingester:**
```yaml
ingester:
  chunk_encoding: snappy
  chunk_idle_period: 30m
  max_chunk_age: 2h
  chunk_target_size: 1572864  # 1.5MB
  lifecycler:
    ring:
      replication_factor: 3  # 3 for production
```

**Querier:**
```yaml
querier:
  max_concurrent: 4
  query_timeout: 1m
```

**Compactor:**
```yaml
compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
```

### Stage 5: Limits Configuration

```yaml
limits_config:
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_streams_per_user: 10000
  max_entries_limit_per_query: 5000
  max_query_length: 721h
  retention_period: 30d
  allow_structured_metadata: true
  volume_enabled: true
```

### Stage 6: Server & Auth

```yaml
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info

auth_enabled: true  # false for single-tenant
```

### Stage 7: OTLP Ingestion (Loki 3.0+)

Native OpenTelemetry ingestion - use `otlphttp` exporter (NOT deprecated `lokiexporter`):

```yaml
limits_config:
  allow_structured_metadata: true
  otlp_config:
    resource_attributes:
      attributes_config:
        - action: index_label  # Low-cardinality only!
          attributes: [service.name, service.namespace, deployment.environment]
        - action: structured_metadata  # High-cardinality
          attributes: [k8s.pod.name, service.instance.id]
```

**Actions:** `index_label` (searchable, low-cardinality), `structured_metadata` (queryable), `drop`

> **⚠️ NEVER use `k8s.pod.name` as index_label** - use structured_metadata instead.

**OTel Collector:**
```yaml
exporters:
  otlphttp:
    endpoint: http://loki:3100/otlp
```

### Stage 8: Caching

```yaml
chunk_store_config:
  chunk_cache_config:
    memcached_client:
      host: memcached-chunks
      timeout: 500ms

query_range:
  cache_results: true
  results_cache:
    cache:
      memcached_client:
        host: memcached-results
```

### Stage 9: Advanced Features

**Pattern Ingester (3.0+):**
```yaml
pattern_ingester:
  enabled: true
```

**Bloom Filters (Experimental, 3.3+):** Only for >75TB/month deployments. Works on structured metadata only. See examples/ for config.

**Time Sharding (3.4+):** For out-of-order ingestion:
```yaml
limits_config:
  shard_streams:
    time_sharding_enabled: true
```

**Thanos Storage (3.4+):** New storage client, opt-in now, default later:
```yaml
storage_config:
  use_thanos_objstore: true
  object_store:
    s3:
      bucket_name: my-bucket
      endpoint: s3.us-west-2.amazonaws.com
```

### Stage 10: Ruler (Alerting)

```yaml
ruler:
  storage:
    type: s3
    s3: { bucket_name: loki-ruler }
  alertmanager_url: http://alertmanager:9093
  enable_api: true
  enable_sharding: true
```

### Stage 11: Loki 3.6 Features

- **Horizontally Scalable Compactor:** `horizontal_scaling_mode: main|wo