bm25-tuning

What this skill does

# BM25 Tuning

## The Formula

```
score(q, d) = sum over q_terms t of
    IDF(t) * ( f(t, d) * (k1 + 1) ) / ( f(t, d) + k1 * (1 - b + b * |d| / avgdl) )
```

- `f(t, d)`: term frequency in document d
- `|d|`: document length in tokens
- `avgdl`: average document length across the corpus
- `k1`: term-frequency saturation (how fast extra occurrences stop helping)
- `b`: length normalization (how much longer documents are penalized)
- `IDF(t)`: inverse document frequency of t

## k1 and b Defaults

| Collection type | k1 | b | Why |
|---|---|---|---|
| Lucene default | 1.2 | 0.75 | Safe general-purpose |
| Short homogeneous docs (titles, tweets) | 1.0-1.2 | 0.3-0.5 | Length already similar; less penalization |
| Long heterogeneous docs (web, manuals) | 1.2-1.5 | 0.75-0.85 | Long docs over-reward frequent terms otherwise |
| Code / logs with rare tokens | 1.5-2.0 | 0.5-0.75 | k1 high so repeated identifiers still accumulate |
| Q&A short passages | 0.8-1.2 | 0.4-0.6 | Passages are near-uniform length |

Tune in a sweep: grid k1 in {0.8, 1.0, 1.2, 1.5, 2.0}, b in {0.3, 0.5, 0.75, 0.9}. Measure NDCG@10 or recall@20 on a gold set.

## Elasticsearch Similarity Config

```json
PUT /docs
{
  "settings": {
    "index": {
      "similarity": {
        "bm25_long": {
          "type": "BM25",
          "k1": 1.5,
          "b": 0.85
        }
      },
      "analysis": {
        "analyzer": {
          "en_custom": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "asciifolding", "english_stop", "english_stemmer"]
          }
        },
        "filter": {
          "english_stop": {"type": "stop", "stopwords": "_english_"},
          "english_stemmer": {"type": "stemmer", "language": "light_english"}
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {"type": "text", "analyzer": "en_custom", "boost": 3.0,
                "similarity": "bm25_long"},
      "body":  {"type": "text", "analyzer": "en_custom",
                "similarity": "bm25_long"}
    }
  }
}
```

## Field Boosts

Boost title over body, headings over paragraphs:

```json
{
  "query": {
    "multi_match": {
      "query": "oauth token refresh 403",
      "type": "best_fields",
      "fields": ["title^3", "headings^2", "body^1"],
      "tie_breaker": 0.3
    }
  }
}
```

`best_fields` returns the highest single field score; `most_fields` sums. `tie_breaker` blends the two. For question-answer ranking, `cross_fields` with a shared analyzer usually beats `best_fields`.

## Language-Specific Analyzers

Non-English corpora fail silently with the default analyzer. Always pick a language-specific one.

### Italian

```json
"it_custom": {
  "type": "custom",
  "tokenizer": "standard",
  "filter": [
    "lowercase",
    "asciifolding",
    "italian_elision",
    "italian_stop",
    "italian_stemmer"
  ]
},
"filter": {
  "italian_elision": {
    "type": "elision",
    "articles_case": true,
    "articles": ["c", "l", "all", "dall", "dell", "nell", "sull", "coll", "pell",
                 "gl", "agl", "dagl", "degl", "negl", "sugl", "un", "m", "t", "s",
                 "v", "d"]
  },
  "italian_stop": {"type": "stop", "stopwords": "_italian_"},
  "italian_stemmer": {"type": "stemmer", "language": "light_italian"}
}
```

Italian needs elision handling (`l'amore` -> `amore`), otherwise queries for "amore" miss documents with the apostrophe form.

### French

```json
"fr_custom": {
  "type": "custom",
  "tokenizer": "standard",
  "filter": [
    "lowercase",
    "asciifolding",
    "french_elision",
    "french_stop",
    "french_stemmer"
  ]
},
"filter": {
  "french_elision": {
    "type": "elision",
    "articles_case": true,
    "articles": ["l", "m", "t", "qu", "n", "s", "j", "d", "c", "jusqu",
                 "quoiqu", "lorsqu", "puisqu"]
  },
  "french_stop": {"type": "stop", "stopwords": "_french_"},
  "french_stemmer": {"type": "stemmer", "language": "light_french"}
}
```

Use `light_french` over `french`; aggressive stemming conflates unrelated roots.

### German

```json
"de_custom": {
  "type": "custom",
  "tokenizer": "standard",
  "filter": [
    "lowercase",
    "german_normalization",
    "german_stop",
    "german_stemmer"
  ]
},
"filter": {
  "german_stop": {"type": "stop", "stopwords": "_german_"},
  "german_stemmer": {"type": "stemmer", "language": "light_german"}
}
```

German compounds (`Bundesausbildungsförderungsgesetz`) need `decompound_token_filter` via an external dictionary — plain stemmers do not split them. Consider `hyphenation_decompounder` with the OpenOffice hyphenation files.

## Stemming vs Lemmatization

| Aspect | Stemming | Lemmatization |
|---|---|---|
| Approach | Strip suffixes by rules | Reduce to dictionary lemma |
| Tool | Snowball / Porter / Lovins | spaCy, Stanza |
| Speed | Microseconds | Milliseconds |
| Accuracy | Crude (runner/running -> run, but also universe/university -> univers) | Correct (better -> good) |
| Storage | Same vocab | Same vocab |
| Production default | Elasticsearch `light_*` stemmers | Only when stem conflation hurts recall |

Most BM25 pipelines stick with stemmers. Lemmatize when you have strong morphology (Finnish, Turkish, Russian) and precision matters.

## Stopword Handling

Default stopword lists remove `the`, `a`, `of`, etc. Two gotchas:

1. Query like "to be or not to be" becomes empty after stopword removal. Use `stop` filter with `remove_trailing=false` or skip stopwords on short queries.
2. Domain-specific words may act as stopwords (`system`, `user` in a software manual). Measure IDF, remove terms with IDF below a threshold as a custom stop set.

```json
"custom_stop": {
  "type": "stop",
  "stopwords": ["system", "user", "module", "click"]
}
```

## Tokenization Gotchas

- `standard` tokenizer splits on Unicode word boundaries — it breaks `error_code_403` into `error`, `code`, `403`. Use `whitespace` tokenizer + `word_delimiter_graph` when identifiers matter.
- URLs / emails: use `uax_url_email` tokenizer if they are keys to your queries.
- Camel-case code tokens: add `word_delimiter_graph` with `generate_word_parts=true`.
- Numbers: BM25 treats `403` and `404` as equally distant from `Forbidden`. Keep them as tokens; rely on exact-term ranking via `constant_score` clauses for known IDs.

```json
"code_analyzer": {
  "type": "custom",
  "tokenizer": "whitespace",
  "filter": [
    "lowercase",
    {
      "type": "word_delimiter_graph",
      "preserve_original": true,
      "split_on_numerics": false,
      "catenate_words": true
    }
  ]
}
```

## Python: rank_bm25 for Prototyping

```python
# pip install rank_bm25 nltk
from rank_bm25 import BM25Okapi, BM25Plus, BM25L
import re

def tokenize(text: str) -> list[str]:
    return re.findall(r"\w+", text.lower())

docs = ["OAuth 2.0 uses refresh tokens.", "PKCE protects public clients.", ...]
tokenized = [tokenize(d) for d in docs]

bm = BM25Okapi(tokenized, k1=1.2, b=0.75)
scores = bm.get_scores(tokenize("token refresh"))
top = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:10]
```

`BM25Plus` (2014 variant) slightly helps on short documents; `BM25L` helps on long ones. All three accept `k1` and `b`.

## Lucene vs Elasticsearch vs rank_bm25

| Engine | Best for | Downsides |
|---|---|---|
| Elasticsearch / OpenSearch | Production, scale, rich analyzers | Cluster ops overhead |
| Tantivy (Rust) / Meilisearch | Single-node high-performance | Smaller analyzer ecosystem |
| Lucene direct (Java) | Embedded JVM apps | Write your own analyzer plumbing |
| rank_bm25 (Python) | Research, tests, single-host | No analyzer pipeline; bring your own |
| Whoosh | Pure Python, offline | Slow and unmaintained |
| Pyserini | BEIR reproduction, anserini-style | Heavy deps |

## When BM25 Alone Beats Vectors

- Legal, medical, financial — exact-term matching on codes / names dominates.
- Short queries with rare identifiers (error codes, SKUs, version numbers).
- Small corpora (< 5k documents