elasticsearch-security-troubleshooting

What this skill does

# Elasticsearch Security Troubleshooting

Diagnose and resolve common Elasticsearch security issues. This skill provides a structured triage workflow for
authentication failures, authorization errors, TLS problems, API key issues, role mapping mismatches, Kibana login
failures, and license-expiry lockouts.

For authentication methods and API key management, see the **elasticsearch-authn** skill. For roles, users, and role
mappings, see the **elasticsearch-authz** skill. For license management, see the **elasticsearch-license** skill.

For diagnostic API endpoints, see [references/api-reference.md](references/api-reference.md).

> **Deployment note:** Diagnostic API availability differs between self-managed, ECH, and Serverless. See
> [Deployment Compatibility](#deployment-compatibility) for details.

## Jobs to Be Done

- Diagnose HTTP 401 authentication failures
- Diagnose HTTP 403 permission denied errors
- Troubleshoot TLS/SSL handshake or certificate errors
- Investigate expired or invalid API keys
- Debug role mappings that do not grant expected roles
- Fix Kibana login failures, redirect loops, or CORS errors
- Recover from a license-expiry lockout
- Determine why a user lacks access to a specific index

## Prerequisites

| Item                   | Description                                                                |
| ---------------------- | -------------------------------------------------------------------------- |
| **Elasticsearch URL**  | Cluster endpoint (e.g. `https://localhost:9200` or a Cloud deployment URL) |
| **Authentication**     | Any valid credentials — even minimal — to reach the cluster                |
| **Cluster privileges** | `monitor` for read-only diagnostics; `manage_security` for fixes           |

Prompt the user for any missing values. If the user cannot authenticate at all, start with
[TLS and Certificate Errors](#tls-and-certificate-errors) or [License Expiry Recovery](#license-expiry-recovery).

## Diagnostic Workflow

Route the symptom to the correct section:

| Symptom                                        | Section                                                       |
| ---------------------------------------------- | ------------------------------------------------------------- |
| HTTP 401, `authentication_exception`           | [Authentication Failures](#authentication-failures-401)       |
| HTTP 403, `security_exception`, access denied  | [Authorization Failures](#authorization-failures-403)         |
| SSL/TLS handshake error, certificate rejected  | [TLS and Certificate Errors](#tls-and-certificate-errors)     |
| API key rejected, expired, or ineffective      | [API Key Issues](#api-key-issues)                             |
| Role mapping not granting expected roles       | [Role Mapping Issues](#role-mapping-issues)                   |
| Kibana login broken, redirect loop, CORS error | [Kibana Authentication Issues](#kibana-authentication-issues) |
| All users locked out, paid features disabled   | [License Expiry Recovery](#license-expiry-recovery)           |

Each section follows a **Gather - Diagnose - Resolve** pattern.

## Diagnostic Toolkit

Use these APIs at the start of any security investigation:

```bash
curl <auth_flags> "${ELASTICSEARCH_URL}/_security/_authenticate"
```

Confirms identity, realm, and roles. If this fails with 401, the problem is authentication.

```bash
curl <auth_flags> "${ELASTICSEARCH_URL}/_xpack"
```

Confirms whether security is enabled (`features.security.enabled`). If security is disabled, all security APIs return
errors.

```bash
curl -X POST "${ELASTICSEARCH_URL}/_security/user/_has_privileges" \
  <auth_flags> \
  -H "Content-Type: application/json" \
  -d '{
    "index": [
      { "names": ["'"${INDEX_PATTERN}"'"], "privileges": ["read"] }
    ]
  }'
```

Tests whether the authenticated user holds specific privileges without requiring `manage_security`.

```bash
curl <auth_flags> "${ELASTICSEARCH_URL}/_license"
```

Check license type and status. An expired paid license disables paid realms and features.

## Authentication Failures (401)

A 401 response means Elasticsearch could not verify the caller's identity.

### Gather

```bash
curl -v <auth_flags> "${ELASTICSEARCH_URL}/_security/_authenticate" 2>&1
```

The `-v` flag shows headers and the response body. Look for:

- `WWW-Authenticate` header — indicates which auth schemes the cluster accepts.
- `authentication_exception` in the response body — the `reason` field describes what failed.

### Diagnose

| Symptom                                            | Likely cause                                    |
| -------------------------------------------------- | ----------------------------------------------- |
| `unable to authenticate user`                      | Wrong username or password                      |
| `unable to authenticate with provided credentials` | Credentials do not match any realm in the chain |
| `user is not enabled`                              | The native user account is disabled             |
| `token is expired`                                 | API key or bearer token has expired             |
| No `WWW-Authenticate` header                       | Security may be disabled; check `GET /_xpack`   |

If the user authenticates via an external realm (LDAP, AD, SAML, OIDC), the realm chain order matters. Elasticsearch
tries realms in configured order and stops at the first match. If a higher-priority realm rejects the credentials before
the intended realm is reached, authentication fails.

### Resolve

| Cause                   | Action                                                                     |
| ----------------------- | -------------------------------------------------------------------------- |
| Wrong credentials       | Verify username/password or API key value. See **elasticsearch-authn**.    |
| Disabled user           | `PUT /_security/user/{name}/_enable`. See **elasticsearch-authz**.         |
| Expired API key         | Create a new API key. See [API Key Issues](#api-key-issues).               |
| Realm chain order       | Check `elasticsearch.yml` realm order (self-managed only).                 |
| Security disabled       | Enable `xpack.security.enabled: true` in `elasticsearch.yml` and restart.  |
| Paid realm after expiry | License expired — see [License Expiry Recovery](#license-expiry-recovery). |

## Authorization Failures (403)

A 403 response means the user is authenticated but lacks the required privileges.

### Gather

Test the specific privileges the operation requires:

```bash
curl -X POST "${ELASTICSEARCH_URL}/_security/user/_has_privileges" \
  <auth_flags> \
  -H "Content-Type: application/json" \
  -d '{
    "index": [
      { "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] }
    ],
    "cluster": ["monitor"]
  }'
```

The response contains a `has_all_requested` boolean and per-resource breakdowns.

Also check the user's effective roles:

```bash
curl <auth_flags> "${ELASTICSEARCH_URL}/_security/_authenticate"
```

Inspect the `roles` array and `authentication_realm` to confirm the user is who you expect.

### Diagnose

| Symptom                                   | Likely cause                                           |
| ----------------------------------------- | ------------------------------------------------------ |
| `has_all_requested: false` for an index   | Role is missing the required index privilege           |
| `has_all_requested: false` for a cluster  | Role is missing the required cluster privilege         |
| User has fewer roles than expected        | Roles array was replaced (not merged) on last update   |
| API key returns 403 on previously allowed | API key privileges are a snapshot — role changes after |
| operation                                 | creation do not propagate to existing keys             |

### Resolve

| Cause                     | Action