elasticsearch

What this skill does

# Elasticsearch

All Elasticsearch interaction is via REST API using `curl`. No SDK or client library required.

## Authentication

Every request needs the cluster URL and an API key:

```bash
# Set these for your session (or export in .env / shell profile)
ES_URL="https://your-cluster.es.cloud.elastic.co:443"
ES_API_KEY="your-base64-api-key"

# All requests follow this pattern:
curl -s "${ES_URL%/}/<endpoint>" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '<json-body>'
```

**API key format:** Base64-encoded `id:api_key` string. Pass as-is in the `Authorization: ApiKey` header.

If the user provides a URL and key, export them as `ES_URL` and `ES_API_KEY` before running commands.

**Important — variable expansion in curl:**
- Always use `$(printenv ES_API_KEY)` instead of `$ES_API_KEY` in curl headers. The `$ES_API_KEY` variable may not expand correctly in the shell, resulting in empty `Authorization` headers and 401 errors.
- Always use `${ES_URL%/}` to strip any trailing slash from the URL, preventing double-slash path issues (e.g., `//_cluster/health`).

## Quick Health Check

```bash
# Cluster health (green/yellow/red) — NOT available on serverless
curl -s "${ES_URL%/}/_cluster/health" -H "Authorization: ApiKey $(printenv ES_API_KEY)" | jq .

# Node stats summary — NOT available on serverless
curl -s "${ES_URL%/}/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent"  \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)"

# Index overview (works on both serverless and traditional)
curl -s "${ES_URL%/}/_cat/indices?v&s=store.size:desc&h=index,health,status,docs.count,store.size" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)"
```

**Serverless Elasticsearch:** If you get `api_not_available_exception` errors, the cluster is running in serverless mode. The following APIs are **not available** in serverless:
- `_cluster/health`, `_cluster/settings`, `_cluster/allocation/explain`, `_cluster/pending_tasks`
- `_cat/nodes`, `_cat/shards`
- `_nodes/hot_threads`, `_nodes/stats`
- ILM APIs (`_ilm/*`)

Use `_cat/indices` and `_search` APIs as the starting point instead — these work everywhere.

## Search (Query DSL)

```bash
# Simple match query
curl -s "${ES_URL%/}/my-index/_search" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match": { "message": "error timeout" } },
    "size": 10
  }' | jq .

# Bool query (must + filter + must_not)
curl -s "${ES_URL%/}/my-index/_search" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "bool": {
        "must": [ { "match": { "message": "error" } } ],
        "filter": [ { "range": { "@timestamp": { "gte": "now-1h" } } } ],
        "must_not": [ { "term": { "level": "debug" } } ]
      }
    },
    "size": 20,
    "sort": [ { "@timestamp": { "order": "desc" } } ]
  }' | jq .
```

For full Query DSL reference (term, terms, range, wildcard, regexp, nested, exists, multi_match, etc.), see [references/query-dsl.md](references/query-dsl.md).

## Index & Document Operations

```bash
# Create index with mappings
curl -s -X PUT "${ES_URL%/}/my-index" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": { "number_of_shards": 1, "number_of_replicas": 1 },
    "mappings": {
      "properties": {
        "message":    { "type": "text" },
        "@timestamp": { "type": "date" },
        "level":      { "type": "keyword" },
        "count":      { "type": "integer" }
      }
    }
  }'

# Index a document (auto-generate ID)
curl -s -X POST "${ES_URL%/}/my-index/_doc" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{ "message": "hello world", "@timestamp": "2026-01-31T12:00:00Z", "level": "info" }'

# Index with specific ID
curl -s -X PUT "${ES_URL%/}/my-index/_doc/doc-123" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{ "message": "specific doc", "level": "warn" }'

# Get document
curl -s "${ES_URL%/}/my-index/_doc/doc-123" -H "Authorization: ApiKey $(printenv ES_API_KEY)" | jq .

# Update document (partial)
curl -s -X POST "${ES_URL%/}/my-index/_update/doc-123" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{ "doc": { "level": "error" } }'

# Delete document
curl -s -X DELETE "${ES_URL%/}/my-index/_doc/doc-123" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)"

# Bulk operations (newline-delimited JSON)
curl -s -X POST "${ES_URL%/}/_bulk" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @- << 'EOF'
{"index":{"_index":"my-index"}}
{"message":"bulk doc 1","level":"info","@timestamp":"2026-01-31T12:00:00Z"}
{"index":{"_index":"my-index"}}
{"message":"bulk doc 2","level":"warn","@timestamp":"2026-01-31T12:01:00Z"}
EOF
```

## Aggregations

```bash
# Terms aggregation (top values)
curl -s "${ES_URL%/}/my-index/_search?size=0" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "aggs": {
      "levels": { "terms": { "field": "level", "size": 10 } }
    }
  }' | jq '.aggregations'

# Date histogram + nested metric
curl -s "${ES_URL%/}/my-index/_search?size=0" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "range": { "@timestamp": { "gte": "now-24h" } } },
    "aggs": {
      "over_time": {
        "date_histogram": { "field": "@timestamp", "fixed_interval": "1h" },
        "aggs": {
          "avg_count": { "avg": { "field": "count" } }
        }
      }
    }
  }' | jq '.aggregations'
```

For more aggregation types (cardinality, percentiles, composite, filters, significant_terms, etc.), see [references/aggregations.md](references/aggregations.md).

## Mappings & Index Management

```bash
# Get mapping
curl -s "${ES_URL%/}/my-index/_mapping" -H "Authorization: ApiKey $(printenv ES_API_KEY)" | jq .

# Add field to existing mapping (mappings are additive — you can't change existing field types)
curl -s -X PUT "${ES_URL%/}/my-index/_mapping" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{ "properties": { "new_field": { "type": "keyword" } } }'

# Reindex (change mappings, rename index, etc.)
curl -s -X POST "${ES_URL%/}/_reindex" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "old-index" },
    "dest":   { "index": "new-index" }
  }'

# Delete index
curl -s -X DELETE "${ES_URL%/}/my-index" -H "Authorization: ApiKey $(printenv ES_API_KEY)"

# Index aliases
curl -s -X POST "${ES_URL%/}/_aliases" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      { "add": { "index": "my-index-v2", "alias": "my-index" } },
      { "remove": { "index": "my-index-v1", "alias": "my-index" } }
    ]
  }'

# Index templates (for time-series / rollover patterns)
curl -s -X PUT "${ES_URL%/}/_index_template/my-template" \
  -H "Authorization: ApiKey $(printenv ES_API_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "index_patterns": ["logs-*"],
    "template": {
      "settings": { "number_of_shards": 1 },
      "mappings": {
        "properties": {
          "message":    { "type": "text" },
          "@timestamp": { "type": "date" }
        }
      }
    }
  }'
```

## Cluster & Troubleshooting

> **Note:** Most APIs in this section are **not available on serverless** Elasticsearch. They only work on self-managed or traditional Elastic Cloud deployments.

```bash
# Allocation explanation (why is a shard unassigned?) — NOT serverless
curl -s "${ES_URL%/}/_cluster/allocation/explain" \
  -H "Auth