brightdata-local-search

What this skill does

# Bright Data Local Search

Run powerful web searches locally using Bright Data's SERP API. This skill sets up the [unfancy-search](https://github.com/yaronbeen/unfancy-search) pipeline — a local search engine with query expansion, multi-engine SERP retrieval, RRF reranking, deduplication, and domain clustering.

**Important: This skill uses the LOCAL version only. Do not use the hosted endpoint.**

## Prerequisites

- [Bright Data](https://brightdata.com) account with SERP API access
- [Anthropic API key](https://console.anthropic.com) (for query expansion, optional)
- Docker (recommended) or Node.js 18+ with pnpm

## Setup

### Step 1: Clone and Configure

```bash
git clone https://github.com/yaronbeen/unfancy-search.git
cd unfancy-search
cp .env.example .env
```

### Step 2: Set Environment Variables

Edit `.env` with your credentials:

```env
BRIGHT_DATA_API_TOKEN=your_brightdata_token
BRIGHT_DATA_SERP_ZONE=serp_api1
ANTHROPIC_API_KEY=your_anthropic_key   # Optional: enables AI query expansion
```

Get your Bright Data token from: https://brightdata.com (SERP API section)

### Step 3: Start the Local Server

**Docker (recommended):**
```bash
docker compose up -d
# Server runs at http://localhost:3000
```

**Node.js:**
```bash
pnpm install
pnpm dev
# Server runs at http://localhost:3000
```

## API Endpoints

All requests go to `http://localhost:3000`:

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/search` | POST | Start a search job |
| `/api/search-status/{jobId}` | GET | Poll for results |
| `/api/baseline` | POST | Trigger baseline collection |
| `/api/baseline-status/{id}` | GET | Poll baseline progress |

## Running a Search

### Step 1: Submit Search

```bash
curl -X POST http://localhost:3000/api/search \
  -H "Content-Type: application/json" \
  -d '{"query": "your search term"}'
```

The response returns a `jobId`.

### Step 2: Poll for Results

```bash
curl http://localhost:3000/api/search-status/{jobId}
```

Poll every 3 seconds until `status` is `"done"`.

### Search Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | string | required | Search query |
| `expand` | boolean | `false` | Enable AI query expansion via Claude |
| `research` | boolean | `false` | Research mode (12 sub-queries for max coverage) |
| `engines` | string[] | all | SERP engines to use |
| `geo` | string | — | Geographic region filter |
| `count` | number | 10 | Max results (up to 10) |
| `includeDomains` | string[] | — | Only include results from these domains |
| `excludeDomains` | string[] | — | Exclude results from these domains |

### Search Modes

- **Basic** (`expand: false`): Single query, fastest, no AI cost
- **Expanded** (`expand: true`): Claude Haiku generates 3 sub-queries for broader coverage
- **Research** (`research: true`): 12 sub-queries for maximum coverage

## Usage Examples

### Basic Search from an Agent

```bash
# Start search
JOB_ID=$(curl -s -X POST http://localhost:3000/api/search \
  -H "Content-Type: application/json" \
  -d '{"query": "best practices for API rate limiting"}' | jq -r '.jobId')

# Poll until done
while true; do
  RESULT=$(curl -s http://localhost:3000/api/search-status/$JOB_ID)
  STATUS=$(echo $RESULT | jq -r '.status')
  if [ "$STATUS" = "done" ]; then
    echo $RESULT | jq '.results'
    break
  fi
  sleep 3
done
```

### Research Mode with Domain Filtering

```bash
curl -X POST http://localhost:3000/api/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "kubernetes scaling strategies",
    "research": true,
    "excludeDomains": ["pinterest.com", "quora.com"]
  }'
```

### Adding Search to an Existing Agent

To give your Claude Code agent search capabilities:

1. Ensure the local server is running (`docker compose up -d` in the unfancy-search directory)
2. Your agent can use `curl` or `fetch` to query `http://localhost:3000/api/search`
3. Parse the ranked results to ground responses with real web data

## Response Format

Results include:
- Ranked URLs with RRF scores
- Domain clustering (grouped by source)
- Cost transparency (per-search expense breakdown)
- Raw and unique result counts
- Search duration

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Server won't start | Verify Docker is running or Node.js 18+ installed |
| No results returned | Check `BRIGHT_DATA_API_TOKEN` is valid and SERP API zone is active |
| Query expansion not working | Verify `ANTHROPIC_API_KEY` is set and valid |
| Slow responses | Disable `expand` mode for faster single-query searches |
| Port 3000 in use | Stop other services or modify the port in docker-compose.yml |