parsing-helper

What this skill does

# Parsing Helper

A guided workflow for creating, testing, and deploying Grok parsing configurations for LimaCharlie adapters. This skill helps you customize how log data is parsed and normalized as it's ingested into LimaCharlie.

---

## LimaCharlie Integration

> **Prerequisites**: Run `/init-lc` to initialize LimaCharlie context.

### LimaCharlie CLI Access

All LimaCharlie operations use the `limacharlie` CLI directly:

```bash
limacharlie <noun> <verb> --oid <oid> --output yaml [flags]
```

For command help and discovery: `limacharlie <command> --ai-help`

### Critical Rules

| Rule | Wrong | Right |
|------|-------|-------|
| **CLI Access** | Call MCP tools or spawn api-executor | Use `Bash("limacharlie ...")` directly |
| **Output Format** | `--output json` | `--output yaml` (more token-efficient) |
| **Filter Output** | Pipe to jq/yq | Use `--filter JMESPATH` to select fields |
| **LCQL Queries** | Write query syntax manually | Use `limacharlie ai generate-query` first |
| **Timestamps** | Calculate epoch values | Use `date +%s` or `date -d '7 days ago' +%s` |
| **OID** | Use org name | Use UUID (call `limacharlie org list` if needed) |

---

> ⚠️ **TIMEZONE NOTE**: Patterns like `SYSLOGTIMESTAMP` (`Dec 16 17:50:04`) lack timezone info. LimaCharlie assumes UTC. This skill automatically detects timezone mismatches by comparing parsed times to current UTC.

## When to Use

Use this skill when:

- **Setting up new log parsing**: Configure Grok patterns for a new log source
- **Troubleshooting parsing issues**: Logs aren't being parsed correctly or fields are missing
- **Modifying field extraction**: Need to add, remove, or change extracted fields
- **Testing parsing changes**: Validate new parsing rules before deploying to production
- **Generating adapter configurations**: Create YAML/CLI config for local USP adapters

Common scenarios:
- "Help me parse these firewall logs"
- "The timestamp isn't being extracted correctly from my syslog"
- "I need to add a new field to my adapter parsing"
- "Generate a Grok pattern for these log samples"
- "Test this parsing configuration before I deploy it"

## What This Skill Does

This skill provides a 5-phase workflow:

1. **Organization Selection** - Select the LimaCharlie organization
2. **Adapter Discovery** - Identify the adapter type and get current configuration
3. **Sample Log Collection** - Get sample logs from the adapter's sensor or user input
4. **Grok Pattern Generation** - Analyze logs and generate appropriate Grok patterns
5. **Validation & Deployment** - Test parsing and apply configuration

### Supported Adapter Types

| Type | Description | Configuration Method |
|------|-------------|---------------------|
| **External Adapter** | Cloud-managed adapters (syslog, webhook, API) | Update via `set_external_adapter` API |
| **Cloud Sensor** | Virtual sensors for cloud platforms (AWS, Azure, GCP, SaaS) | Update via `set_cloud_sensor` API |
| **One-off/USP Adapter** | On-prem adapters with local configuration | Output YAML/CLI config for user |

## Required Information

Before starting, gather:

- **Organization**: The LimaCharlie org where the adapter is configured
- **Adapter Type**: External Adapter, Cloud Sensor, or One-off/USP Adapter
- **Adapter Name**: For cloud-managed adapters, the name of the existing adapter
- **Sample Logs**: Either from the adapter's sensor or user-provided samples

## How to Use

### Phase 1: Organization Selection

Get the list of available organizations:

```bash
limacharlie org list --output yaml
```

Use `AskUserQuestion` to let the user select an organization if multiple are available.

### Phase 2: Adapter Discovery & Type Selection

Ask the user which adapter type they're working with using `AskUserQuestion`:

**Option A: External Adapter** (cloud-managed syslog, webhook, API)

1. List available external adapters:
```bash
limacharlie external-adapter list --oid <SELECTED_ORG_ID> --output yaml
```

2. Get the selected adapter's current configuration:
```bash
limacharlie external-adapter get --key <ADAPTER_NAME> --oid <SELECTED_ORG_ID> --output yaml
```

**Option B: Cloud Sensor** (AWS, Azure, GCP, SaaS)

1. List available cloud sensors:
```bash
limacharlie cloud-adapter list --oid <SELECTED_ORG_ID> --output yaml
```

2. Get the selected cloud sensor's current configuration:
```bash
limacharlie cloud-adapter get --key <SENSOR_NAME> --oid <SELECTED_ORG_ID> --output yaml
```

**Option C: One-off/USP Adapter** (local adapter, no cloud config)

No API calls needed. Proceed directly to sample log collection.

### Phase 3: Get Sample Log Data

**Option A: Query from adapter's sensor** (for cloud-managed adapters)

If the adapter is already ingesting data, you can fetch sample events:

1. Find the sensor by IID (Installation ID from the adapter's installation key):
```bash
limacharlie sensor list --oid <SELECTED_ORG_ID> --filter "[?iid=='<IID>']" --output yaml
```

2. Get recent events from the sensor:

**IMPORTANT**: Always calculate timestamps dynamically using Bash FIRST:

```bash
# Run this to get current epoch timestamps for the last hour
start=$(date -d '1 hour ago' +%s) && end=$(date +%s) && echo "start=$start end=$end"
```

Then use the actual output values (e.g., `start=1764805928 end=1764809528`) directly in the CLI call - do NOT use placeholder values:

```bash
limacharlie search run --query "event sid = '<SENSOR_ID>'" --start $start --end $end --oid <SELECTED_ORG_ID> --filter "[:10]" --output yaml
```

3. Extract the raw log content from the events for analysis.

**Option B: User provides sample logs**

If no data is available or for one-off adapters, ask the user to paste sample log lines.

Use `AskUserQuestion` or direct text input to collect sample logs.

### Recognizing Unparsed Events

When fetching events from a sensor, **unparsed logs** appear as:
- `event_type: "unknown_event"` in the routing
- Event payload contains only a `text` field with the raw log line

Example of an unparsed event:
```json
{
  "event": {
    "text": "2025-12-03 16:41:19 status installed shared-mime-info:amd64 2.2-1"
  },
  "routing": {
    "event_type": "unknown_event",
    "hostname": "my-adapter"
  }
}
```

This indicates no Grok parsing is configured for the adapter. The raw content in the `text` field is what you need to analyze and create a parsing pattern for.

### Phase 4: Generate Grok Pattern

Analyze the sample logs and generate an appropriate Grok pattern.

**Common Grok Patterns:**

| Pattern | Description | Example Match |
|---------|-------------|---------------|
| `%{TIMESTAMP_ISO8601:timestamp}` | ISO 8601 timestamp (YYYY-MM-DD) | `2024-01-15T12:30:45Z` or `2024-01-15 12:30:45` |
| `%{DATESTAMP:timestamp}` | US/EU date format (MM-DD-YY) | `01-15-24 12:30:45` |
| `%{SYSLOGTIMESTAMP:date}` | Syslog timestamp | `Jan 15 12:30:45` |
| `%{IP:ip_address}` | IPv4/IPv6 address | `192.168.1.100` |
| `%{HOSTNAME:host}` | Hostname | `server01.example.com` |
| `%{NUMBER:value}` | Numeric value | `12345` |
| `%{INT:count}` | Integer | `42` |
| `%{WORD:action}` | Single word | `ACCEPT` |
| `%{DATA:field}` | Non-greedy match (up to delimiter) | `any text` |
| `%{GREEDYDATA:message}` | Greedy match (rest of line) | `everything else` |
| `%{LOGLEVEL:level}` | Log level | `ERROR`, `WARN`, `INFO` |
| `%{POSINT:port}` | Positive integer | `443` |

> **Timestamp Pattern Warning**: Do NOT use `%{DATESTAMP}` for `YYYY-MM-DD` format logs - it will misparse the year (treating `2025-12-03` as day=25, month=12, year=03). Use `%{TIMESTAMP_ISO8601}` instead for any `YYYY-MM-DD` formatted timestamps.

**IMPORTANT - Grok Field Name for Text Platform:**

For text platform adapters, always use `message` as the key in `parsing_grok`:

```yaml
parsing_grok:
  message: '%{PATTERN:field} ...'  # Always use "message" as the key, NOT "text"
```

**Building the Mapping Configuration:**

```yaml
client_options:
  mapping:
    parsing_grok:
      message: '%{SYSLOGTIMESTAMP:date} %{HOSTNAME:host}