web-search-plus

What this skill does

# Web Search Plus

Multi-provider web search with **Intelligent Auto-Routing**: Serper (Google), Tavily (Research), Exa (Neural).

**NEW in v2.3.0**: Interactive setup wizard! Run `python3 scripts/setup.py` for guided configuration.

**NEW in v2.2.5**: Automatic error fallback — if one provider fails (rate limit, timeout, etc.), automatically tries the next provider in priority order!

---

## 🚀 First Run (Setup Wizard)

New to web-search-plus? The interactive setup wizard guides you through configuration:

```bash
python3 scripts/setup.py
```

The wizard will:
1. **Explain each provider** — What they're best for, free tier limits, signup links
2. **Ask which providers to enable** — You can use 1, 2, or all 3
3. **Collect API keys** — Keys are stored locally in `config.json` (gitignored)
4. **Configure defaults** — Default provider, auto-routing, result count

### What Each Provider Is Best For

| Provider | Best For | Free Tier |
|----------|----------|-----------|
| **Serper** | Google results, shopping, prices, local businesses, news | 2,500/month |
| **Tavily** | Research questions, explanations, academic, full-page content | 1,000/month |
| **Exa** | Semantic search, "similar to X", startup discovery, papers | 1,000/month |

### Reconfigure Anytime

```bash
python3 scripts/setup.py --reset
```

---

## 🔑 API Keys Setup (Manual)

**NEW in v2.2.0**: The script **auto-loads** API keys from `.env` in the skill directory!

### Quick Setup

**Option A: .env file** (recommended)
```bash
# /path/to/skills/web-search-plus/.env
export SERPER_API_KEY="your-key"   # https://serper.dev
export TAVILY_API_KEY="your-key"   # https://tavily.com  
export EXA_API_KEY="your-key"      # https://exa.ai
```

**Option B: config.json** (NEW in v2.2.1)
```bash
# Copy the example config
cp config.example.json config.json
```
Then add your keys:
```json
{
  "serper": { "api_key": "your-serper-key" },
  "tavily": { "api_key": "your-tavily-key" },
  "exa": { "api_key": "your-exa-key" }
}
```
⚠️ `config.json` is gitignored — your keys stay safe!

Just run — keys load automatically:
```bash
python3 scripts/search.py -q "your query"
# No need for 'source .env' anymore! ✨
```

**Priority:** config.json > .env > environment variable

### Get Free API Keys

| Provider | Free Tier | Sign Up |
|----------|-----------|---------|
| Serper | 2,500 queries/mo | https://serper.dev |
| Tavily | 1,000 queries/mo | https://tavily.com |
| Exa | 1,000 queries/mo | https://exa.ai |

---

## ⚠️ Don't Modify Core OpenClaw Config

**Tavily, Serper, and Exa are NOT core OpenClaw providers.**

❌ **DON'T** add to `~/.openclaw/openclaw.json`:
```json
"tools": { "web": { "search": { "provider": "tavily" }}}  // WRONG!
```

✅ **DO** use this skill's scripts — keys auto-load from `.env`

Core OpenClaw only supports `brave` as the built-in web search provider. This skill adds Serper, Tavily, and Exa as **additional** options via its own scripts.

---

## 🧠 Intelligent Auto-Routing

No need to choose a provider — just search! The skill uses **multi-signal analysis** to understand your query intent:

```bash
# These queries are intelligently routed with confidence scoring:
python3 scripts/search.py -q "how much does iPhone 16 cost"     # → Serper (68% MEDIUM)
python3 scripts/search.py -q "how does quantum entanglement work"  # → Tavily (86% HIGH)
python3 scripts/search.py -q "startups similar to Notion"       # → Exa (76% HIGH)
python3 scripts/search.py -q "MacBook Pro M3 specs review"      # → Serper (70% HIGH)
python3 scripts/search.py -q "explain pros and cons of React"   # → Tavily (85% HIGH)
python3 scripts/search.py -q "companies like stripe.com"        # → Exa (100% HIGH)
```

### How It Works

The routing engine analyzes multiple signals:

#### 🛒 Shopping Intent → Serper
| Signal Type | Examples | Weight |
|-------------|----------|--------|
| Price patterns | "how much", "price of", "cost of" | HIGH |
| Purchase intent | "buy", "purchase", "order", "where to buy" | HIGH |
| Deal signals | "deal", "discount", "cheap", "best price" | MEDIUM |
| Product + Brand | "iPhone 16", "Sony headphones" + specs/review | HIGH |
| Local business | "near me", "restaurants", "hotels" | HIGH |

#### 📚 Research Intent → Tavily
| Signal Type | Examples | Weight |
|-------------|----------|--------|
| Explanation | "how does", "why does", "explain", "what is" | HIGH |
| Analysis | "compare", "pros and cons", "difference between" | HIGH |
| Learning | "tutorial", "guide", "understand", "learn" | MEDIUM |
| Depth | "in-depth", "comprehensive", "detailed" | MEDIUM |
| Complex queries | Long, multi-clause questions | BONUS |

#### 🔍 Discovery Intent → Exa
| Signal Type | Examples | Weight |
|-------------|----------|--------|
| Similarity | "similar to", "alternatives to", "competitors" | VERY HIGH |
| Company discovery | "companies like", "startups doing", "who else" | HIGH |
| URL detection | Any URL or domain (stripe.com) | VERY HIGH |
| Academic | "arxiv", "research papers", "github projects" | HIGH |
| Funding | "Series A", "YC", "funded startup" | HIGH |

### Confidence Scoring

Every routing decision includes a confidence level:

| Confidence | Level | Meaning |
|------------|-------|---------|
| 70-100% | **HIGH** | Strong signal match, very reliable |
| 40-69% | **MEDIUM** | Good match, should work well |
| 0-39% | **LOW** | Ambiguous query, using fallback |

### Debug Routing Decisions

See the full analysis:

```bash
python3 scripts/search.py --explain-routing -q "how much does iPhone 16 Pro cost"
```

Output:
```json
{
  "query": "how much does iPhone 16 Pro cost",
  "routing_decision": {
    "provider": "serper",
    "confidence": 0.68,
    "confidence_level": "medium",
    "reason": "moderate_confidence_match"
  },
  "scores": {"serper": 7.0, "tavily": 0.0, "exa": 0.0},
  "top_signals": [
    {"matched": "how much", "weight": 4.0},
    {"matched": "brand + product detected", "weight": 3.0}
  ],
  "query_analysis": {
    "word_count": 7,
    "is_complex": false,
    "has_url": null,
    "recency_focused": false
  }
}
```

---

## 🔍 When to Use This Skill vs Built-in Brave Search

### Use **Built-in Brave Search** when:
- ✅ General web searches (news, info, questions)
- ✅ Privacy is important
- ✅ Quick lookups without specific requirements

### Use **web-search-plus** when:

#### → **Serper** (Google results):
- 🛍️ **Product specs, prices, shopping** - "Compare iPhone 16 vs Samsung S24"
- 📍 **Local businesses, places** - "Best pizza in Berlin"
- 🎯 **"Google it"** - Explicitly wants Google results
- 📰 **Shopping/images/news** - `--type shopping/images/news`
- 🏆 **Knowledge Graph** - Structured info (prices, ratings, etc.)

#### → **Tavily** (AI-optimized research):
- 📚 **Research questions** - "How does quantum computing work?"
- 🔬 **Deep dives** - Complex multi-part questions
- 📄 **Full page content** - Not just snippets (`--raw-content`)
- 🎓 **Academic research** - Synthesized answers
- 🔒 **Domain filtering** - `--include-domains` for trusted sources

#### → **Exa** (Neural semantic search):
- 🔗 **Similar pages** - "Sites like OpenAI.com" (`--similar-url`)
- 🏢 **Company discovery** - "AI companies like Anthropic"
- 📝 **Research papers** - `--category "research paper"`
- 💻 **GitHub projects** - `--category github`
- 📅 **Date-specific** - `--start-date` / `--end-date`

---

## Provider Comparison

| Feature | Serper | Tavily | Exa |
|---------|:------:|:------:|:---:|
| Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ |
| Factual Accuracy | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
| Semantic Understanding | ⭐ | ⭐⭐ | ⭐⭐⭐ |
| Research Quality | ⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
| Full Page Content | ✗ | ✓ | ✓ |
| Shopping/Local | ✓ | ✗ | ✗ |
| Similar Pages | ✗ | ✗ | ✓ |
| Knowledge Graph | ✓ | ✗ | ✗ |

---

## Usage Examples

### Auto-Routed (Recommended)

```bash
python3 scripts/search.py -q "iPhone 16 Pro Max price"          # → Serper
python3 scripts/search.py -q "how does HTTPS encryption work"   # → Tavily
python3 scripts/search.py -q "startups simil