memory-lancedb-pro

What this skill does

# memory-lancedb-pro

Production-grade long-term memory system (v1.1.0-beta.8) for OpenClaw AI agents. Provides persistent, intelligent memory storage using LanceDB with hybrid vector + BM25 retrieval, LLM-powered Smart Extraction, Weibull decay lifecycle, and multi-scope isolation.

For full technical details (thresholds, formulas, database schema, source file map), see `references/full-reference.md`.

---

## Applying the Optimal Config (Step-by-Step Workflow)

When the user says "help me enable the best config", "apply optimal configuration", or similar, follow this exact procedure:

### Step 1 — Present configuration plans and let user choose

Present these three plans in a clear comparison, then ask the user to pick one:

---

**Plan A — 🏆 Full Power (Best Quality)**
- Embedding: Jina `jina-embeddings-v5-text-small` (task-aware, 1024-dim)
- Reranker: Jina `jina-reranker-v3` (cross-encoder, same key)
- LLM: OpenAI `gpt-4o-mini` (Smart Extraction)
- Keys needed: `JINA_API_KEY` + `OPENAI_API_KEY`
- Get keys: Jina → https://jina.ai/api-key · OpenAI → https://platform.openai.com/api-keys
- Cost: Both paid (Jina has free tier with limited quota)
- Best for: Production deployments, highest retrieval quality

**Plan B — 💰 Budget (Free Reranker)**
- Embedding: Jina `jina-embeddings-v5-text-small`
- Reranker: SiliconFlow `BAAI/bge-reranker-v2-m3` (free tier available)
- LLM: OpenAI `gpt-4o-mini`
- Keys needed: `JINA_API_KEY` + `SILICONFLOW_API_KEY` + `OPENAI_API_KEY`
- Get keys: Jina → https://jina.ai/api-key · SiliconFlow → https://cloud.siliconflow.cn/account/ak · OpenAI → https://platform.openai.com/api-keys
- Cost: Jina embedding paid, SiliconFlow reranker free tier, OpenAI paid
- Best for: Cost-sensitive deployments that still want reranking

**Plan C — 🟢 Simple (OpenAI Only)**
- Embedding: OpenAI `text-embedding-3-small`
- Reranker: None (vector+BM25 fusion only, no cross-encoder)
- LLM: OpenAI `gpt-4o-mini`
- Keys needed: `OPENAI_API_KEY` only
- Get key: https://platform.openai.com/api-keys
- Cost: OpenAI paid only
- Best for: Users who already have OpenAI and want minimal setup

**Plan D — 🖥️ Fully Local (Ollama, No API Keys)**
- Embedding: Ollama `mxbai-embed-large` (1024-dim, recommended) or `nomic-embed-text:v1.5` (768-dim, lighter)
- Reranker: **None** — Ollama has no cross-encoder reranker; retrieval uses vector+BM25 fusion only
- LLM: Ollama via OpenAI-compatible endpoint — recommended models with reliable JSON/structured output:
  - `qwen3:8b` (**recommended** — best JSON output, native structured output, ~5.2GB)
  - `qwen3:14b` (better quality, ~9GB, needs 16GB VRAM)
  - `llama4:scout` (multimodal MoE, 10M ctx, ~12GB)
  - `mistral-small3.2` (24B, 128K ctx, excellent instruction following, ~15GB)
  - `mistral-nemo` (12B, 128K ctx, efficient, ~7GB)
- Keys needed: **None** — fully local, no external API calls
- Prerequisites:
  - Ollama installed: https://ollama.com/download
  - Models pulled (see Step 5 below)
  - Ollama running: macOS = launch the app from Applications; Linux = `systemctl start ollama` or `ollama serve`
- Cost: Free (hardware only)
- RAM requirements: mxbai-embed-large ~670MB; qwen3:8b ~5.2GB; qwen3:14b ~9GB; llama4:scout ~12GB; mistral-small3.2 ~15GB
- Trade-offs: No cross-encoder reranking = lower retrieval precision than Plans A/B; Smart Extraction quality depends on local LLM — if extraction produces garbage, set `"smartExtraction": false`
- Best for: Privacy-sensitive deployments, air-gapped environments, zero API cost

---

After user selects a plan, ask in one message:
1. Please provide the required API key(s) for your chosen plan (paste directly, or say "already set as env vars")
2. Are the env vars already set in your OpenClaw Gateway process? (If unsure, answer No)
3. Where is your `openclaw.json`? (Skip if you want me to find it automatically)

If the user already stated their provider/keys in context, skip asking and proceed.

**Do NOT proceed to Step 2 until API keys have been collected and verified (Step 2 below).**

### Step 2 — Verify API Keys (MANDATORY — do not skip)

**Run ALL key checks for the chosen plan before touching any config.** If any check fails, STOP and tell the user which key failed and why. Do not proceed to Step 3.

**Plan A / Plan B — Jina embedding check:**
```bash
curl -s -o /dev/null -w "%{http_code}" \
  https://api.jina.ai/v1/embeddings \
  -H "Authorization: Bearer <JINA_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model":"jina-embeddings-v5-text-small","input":["test"]}'
```

**Plan A / B / C — OpenAI check:**
```bash
curl -s -o /dev/null -w "%{http_code}" \
  https://api.openai.com/v1/models \
  -H "Authorization: Bearer <OPENAI_API_KEY>"
```

**Plan B — SiliconFlow reranker check:**
```bash
curl -s -o /dev/null -w "%{http_code}" \
  https://api.siliconflow.com/v1/rerank \
  -H "Authorization: Bearer <SILICONFLOW_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model":"BAAI/bge-reranker-v2-m3","query":"test","documents":["test doc"]}'
```

**Plan D — Ollama check:**
```bash
curl -s -o /dev/null -w "%{http_code}" http://localhost:11434/api/tags
```

**Interpret results:**

| HTTP code | Meaning | Action |
|-----------|---------|--------|
| `200` / `201` | Key valid, quota available | ✅ Continue |
| `401` / `403` | Invalid or expired key | ❌ STOP — ask user to check key |
| `402` | Payment required / no credits | ❌ STOP — ask user to top up account |
| `429` | Rate limited or quota exceeded | ❌ STOP — ask user to check billing/quota |
| `000` / connection refused | Service unreachable | ❌ STOP — ask user to check network / Ollama running |

**If any check fails:** Tell the user exactly which provider failed, the HTTP code received, and what to fix. **Do not proceed with installation until all required keys pass their checks.**

If the user says keys are set as env vars in the gateway process, run checks using `${VAR_NAME}` substituted inline or ask them to paste the key temporarily for verification.

### Step 3 — Find openclaw.json

Check these locations in order:
```bash
# Most common locations
ls ~/.openclaw/openclaw.json
ls ~/openclaw.json
# Ask the gateway where it's reading config from
openclaw config get --show-path 2>/dev/null || echo "not found"
```

If not found, ask the user for the path.

### Step 4 — Read current config

```bash
# Read and display current plugins config before changing anything
openclaw config get plugins.entries.memory-lancedb-pro 2>/dev/null
openclaw config get plugins.slots.memory 2>/dev/null
```

**Check what already exists** — never blindly overwrite existing settings.

### Step 5 — Build the merged config based on chosen plan

Use the config block for the chosen plan. Substitute actual API keys inline if the user provided them directly; keep `${ENV_VAR}` syntax if they confirmed env vars are set in the gateway process.

**Plan A config (`plugins.entries.memory-lancedb-pro.config`):**
```json
{
  "embedding": {
    "apiKey": "${JINA_API_KEY}",
    "model": "jina-embeddings-v5-text-small",
    "baseURL": "https://api.jina.ai/v1",
    "dimensions": 1024,
    "taskQuery": "retrieval.query",
    "taskPassage": "retrieval.passage",
    "normalized": true
  },
  "autoCapture": true,
  "autoRecall": true,
  "captureAssistant": false,
  "smartExtraction": true,
  "extractMinMessages": 2,
  "extractMaxChars": 8000,
  "llm": {
    "apiKey": "${OPENAI_API_KEY}",
    "model": "gpt-4o-mini",
    "baseURL": "https://api.openai.com/v1"
  },
  "retrieval": {
    "mode": "hybrid",
    "vectorWeight": 0.7,
    "bm25Weight": 0.3,
    "rerank": "cross-encoder",
    "rerankProvider": "jina",
    "rerankModel": "jina-reranker-v3",
    "rerankEndpoint": "https://api.jina.ai/v1/rerank",
    "rerankApiKey": "${JINA_API_KEY}",
    "candidatePoolSize": 12,
    "minScore": 0.6,
    "hardMinScore": 0.62,
    "filterNoise": true
  },
  "sessionMemory": { "enabled": false }
}
```

**Plan B config:**
```json
{
  "embedding": {
    "apiKey": "${JINA_