utils:tokf-filter

What this skill does

# tokf Filter Authoring

You are an expert at writing tokf filter files. tokf is a config-driven CLI that compresses command output before it reaches an LLM context. Filters are TOML files that define how to process a command's output.

When the user asks you to create or modify a filter, follow this guide exactly. Produce valid, idiomatic TOML that matches the schema described below.

---

## Section 1 — What a Filter File Is

A filter file is a TOML file that describes:
- Which command(s) it applies to (`command`)
- How to transform the raw output (steps, applied in a fixed order)
- What to emit on success vs. failure

Filters live in three places, searched in priority order:

1. `.tokf/filters/` — project-local (repo-level overrides)
2. `~/.config/tokf/filters/` — user-level overrides
3. Built-in library (embedded in the tokf binary)

First match wins. Use `tokf which "cargo test"` to see which filter would activate for a given command.

---

## Section 2 — Processing Order

Steps execute in this fixed order — **do not rearrange them**:

1. **`match_output`** — whole-output substring checks; if matched, short-circuits the entire pipeline and emits immediately
2. **`[[replace]]`** — per-line regex transforms applied to every line, in array order
3. **`strip_ansi` / `trim_lines`** — per-line cleanup (ANSI stripping, whitespace trimming)
4. **`skip` / `keep`** — line-level filtering (drop or retain lines by regex)
5. **`dedup` / `dedup_window`** — collapse duplicate consecutive lines
6. **`lua_script`** — Luau escape hatch; runs after dedup, before section/parse
7. **`[[section]]` OR `[parse]`** — structured extraction (these are mutually exclusive; section is a state machine, parse is a declarative grouper)
8. **`[[chunk]]`** — block-based structured extraction with per-block aggregation, grouping, and tree output (runs on raw output, alongside sections)
9. **Exit-code branch** — `[on_success]` or `[on_failure]` depending on exit code
10. **`[fallback]`** — if neither `on_success` nor `on_failure` produced output
11. **`strip_empty_lines` / `collapse_empty_lines`** — post-processing cleanup on the final output

Within `[on_success]` and `[on_failure]`, fields are processed as:
- `head` / `tail` → trim lines
- `skip` / `extract` → further filter
- `aggregate` → reduce collected sections
- `output` → final template render

---

## Section 3 — Top-Level Fields Reference

| Field | Type | Default | Description |
|---|---|---|---|
| `command` | string or array of strings | required | Command pattern(s) to match. Supports `*` wildcard. |
| `run` | string | (same as command) | Override the actual command executed. Use `{args}` to forward arguments. |
| `match_output` | array of tables | `[]` | Whole-output checks. Short-circuit on first match. |
| `[[replace]]` | array of tables | `[]` | Per-line regex replacements, in order. |
| `skip` | array of strings (regex) | `[]` | Drop lines matching any regex. |
| `keep` | array of strings (regex) | `[]` | Retain only lines matching any regex. (Inverse of skip.) |
| `dedup` | bool | `false` | Collapse consecutive identical lines. |
| `dedup_window` | integer | `0` (off) | Dedup within a sliding window of N lines. |
| `strip_ansi` | bool | `false` | Strip ANSI escape sequences before skip/keep. |
| `trim_lines` | bool | `false` | Trim leading/trailing whitespace from each line. |
| `lua_script` | table | (absent) | Luau escape hatch. |
| `[[section]]` | array of tables | `[]` | State-machine section collectors. |
| `[[chunk]]` | array of tables | `[]` | Block-based structured extraction with per-block aggregation and grouping. |
| `[parse]` | table | (absent) | Declarative structured parser (branch + group). |
| `[on_success]` | table | (absent) | Output branch for exit code 0. |
| `[on_failure]` | table | (absent) | Output branch for non-zero exit. |
| `[output]` | table | (absent) | Top-level output template (used by `[parse]`). |
| `[fallback]` | table | (absent) | Fallback when no branch matched. |
| `strip_empty_lines` | bool | `false` | Remove all blank lines from the final output. |
| `collapse_empty_lines` | bool | `false` | Collapse consecutive blank lines into one. |
| `show_history_hint` | bool | `false` | Append a hint line after filtered output pointing to the full output in history. |
| `[[variant]]` | array of tables | `[]` | Context-aware delegation to specialized child filters. |

---

## Section 4 — Step Types

### 4.1 `match_output` — Whole-Output Short-Circuit

Check the entire raw output for a substring. If matched, emit a fixed string and stop — no further processing.

```toml
match_output = [
  { contains = "Everything up-to-date", output = "ok (up-to-date)" },
  { contains = "rejected", output = "✗ push rejected (try pulling first)" },
]
```

- `contains`: literal substring to search for (case-sensitive)
- `output`: string to emit if matched
- `{line_containing}` template variable: the first line that contains the substring

```toml
match_output = [
  { contains = "error", output = "Error on: {line_containing}" },
]
```

**When to use**: for well-known one-liner outcomes that make the rest of filtering irrelevant (e.g., "already up to date", "nothing to push", "authentication failed").

---

### 4.2 `[[replace]]` — Per-Line Regex Transforms

Applied to every line, in array order, before skip/keep. Use to reformat noisy lines.

```toml
[[replace]]
pattern = '^(\S+)\s+\S+\s+(\S+)\s+(\S+)'
output = "{1}: {2} → {3}"

[[replace]]
pattern = '^\s+Compiling (\S+) v(\S+)'
output = "compiling {1}@{2}"
```

- `pattern`: Rust regex (RE2 syntax, no lookaheads)
- `output`: template with `{1}`, `{2}`, … for capture groups; `{0}` is the full match
- If the pattern doesn't match a line, that line passes through unchanged
- Invalid patterns are silently skipped at runtime

**When to use**: when a line contains useful information but in a verbose format — reformat it rather than dropping it.

---

### 4.3 `skip` / `keep` — Line Filtering

`skip` drops lines matching any regex. `keep` retains only lines matching any regex. They compose:

```toml
skip = [
  "^\\s*Compiling ",
  "^\\s*Downloading ",
  "^\\s*$",
]

keep = ["^error", "^warning"]
```

- Both are arrays of regex strings
- Applied after `[[replace]]`
- `skip` is checked first, then `keep`
- A line must pass both: not skipped, and (if keep is non-empty) matching keep

**When to use**: `skip` for removing known noise patterns; `keep` for allow-listing (e.g., keep only lines that start with `error` or `warning`).

Also available inside `[on_success]` and `[on_failure]` for branch-level filtering.

---

### 4.4 `dedup` / `dedup_window` — Deduplication

```toml
dedup = true           # collapse consecutive identical lines
dedup_window = 10      # dedup within a 10-line sliding window
```

- `dedup = true`: removes consecutive duplicate lines (like `uniq`)
- `dedup_window = N`: deduplicates within a sliding window of N lines (catches near-consecutive repeats)
- They are independent; you can use both

**When to use**: for commands that emit repetitive progress lines (e.g., `npm install` printing the same package multiple times, spinner frames, repeated warnings).

---

### 4.5 `lua_script` — Luau Escape Hatch

For logic that pure TOML cannot express: numeric math, multi-line lookahead, conditional branching.

```toml
[lua_script]
lang = "luau"
source = '''
if exit_code == 0 then
    return "passed"
else
    local msg = output:match("Error: (.+)") or "unknown error"
    return "FAILED: " .. msg
end
'''
```

Or load the script from an external file:

```toml
[lua_script]
lang = "luau"
file = "scripts/my-filter.luau"
```

The `file` path resolves relative to the current working directory. Exactly one of `source` or `file` must be set.

**Globals available**:
- `output` (string): the full output after skip/keep/dedup
- `exit_code` (integer): the command's exit code
- `args` (table of strings): the arguments passed to the command

**Return semantics**:
- Retur