actionbook-scraper

What this skill does

# Actionbook Scraper Skill

## ⚠️ CRITICAL: Two-Part Verification

**Every generated script MUST pass BOTH checks:**

| Check | What to Verify | Failure Example |
|-------|----------------|-----------------|
| **Part 1: Script Runs** | No errors, no timeouts | `Selector not found` |
| **Part 2: Data Correct** | Content matches expected | Extracted "Click to expand" instead of name |

```
┌─────────────────────────────────────────────────────┐
│   1. Generate Script                                │
│          ↓                                          │
│   2. Execute Script                                 │
│          ↓                                          │
│   3. Check Part 1: Script runs without errors?      │
│          ↓                                          │
│   4. Check Part 2: Data content is correct?         │
│      - Not empty                                    │
│      - Not placeholder text ("Loading...")          │
│      - Not UI text ("Click to expand")              │
│      - Fields mapped correctly                      │
│          ↓                                          │
│      ┌───┴───┐                                      │
│   BOTH Pass  Either Fails                           │
│      │           │                                  │
│      │           ↓                                  │
│      │       Is it Actionbook data issue?           │
│      │           │                                  │
│      │       ┌───┴───┐                              │
│      │      Yes      No                             │
│      │       │       │                              │
│      │       ↓       ↓                              │
│      │    Log to   Fix script                       │
│      │    .actionbook-issues.log                    │
│      │       │       │                              │
│      │       └───┬───┘                              │
│      │           ↓                                  │
│      │       Retry (max 3x)                         │
│      ↓                                              │
│   Output Script                                     │
└─────────────────────────────────────────────────────┘
```

## Default Output Format

```
/actionbook-scraper:generate <url>
```

**DEFAULT = agent-browser script (bash commands)**

```bash
agent-browser open "https://example.com"
agent-browser scroll down 2000
agent-browser get text ".selector"
agent-browser close
```

## With --standalone Flag

```
/actionbook-scraper:generate <url> --standalone
```

**Output = Playwright JavaScript code**

---

## Verification Requirements

### Two-Part Verification

Every generated script must pass BOTH checks:

| Check | What to Verify | Failure Action |
|-------|---------------|----------------|
| **1. Script Runs** | No errors, no timeouts | Fix syntax/selector errors |
| **2. Data Correct** | Content matches expected fields | Fix extraction logic |

### Part 1: Script Execution Check

- No runtime errors
- No timeout errors
- Browser closes properly

### Part 2: Data Content Check (CRITICAL)

**Verify extracted data matches the expected structure:**

```
Expected: Company name, description, website, year founded
Actual:   "Click to expand", "Loading...", empty strings

→ FAIL: Data content incorrect, need to fix extraction logic
```

**Data validation rules:**

| Rule | Example Failure | Fix |
|------|-----------------|-----|
| Fields not empty | `name: ""` | Check selector targets correct element |
| No placeholder text | `name: "Loading..."` | Add wait for dynamic content |
| No UI text | `name: "Click to expand"` | Extract after expanding, not button text |
| Correct data type | `year: "View Details"` | Wrong selector, fix field mapping |
| Reasonable count | Expected ~100, got 3 | Add scroll/pagination handling |

### For agent-browser Scripts

1. **Execute the generated commands**
2. **Check script runs without errors**
3. **Check data content is correct:**
   - Fields match expected structure
   - Values are actual data, not UI text
   - Count is reasonable
4. **If failed:**
   - Analyze what's wrong (script error vs data error)
   - Fix selector, wait logic, or extraction
   - Re-execute
5. **If success:**
   - Output the verified script
   - Show data preview with field validation

### For Playwright Scripts (--standalone)

1. **Write script to temp file**
2. **Run with `node script.js`**
3. **Check script runs without errors**
4. **Check output data is correct:**
   - JSON structure matches expected fields
   - Values contain actual data
   - Count matches expected range
5. **If failed:**
   - Analyze error type
   - Fix script
   - Re-run
6. **If success:**
   - Output the verified script

## Architecture Overview

```
/generate <url>              → OUTPUT: agent-browser bash commands
/generate <url> --standalone → OUTPUT: Playwright .js file
```

```
┌─────────────────────────────────────────────────────────────┐
│                   /generate <url>                           │
│                                                             │
│   1. Search Actionbook → get selectors                      │
│   2. Generate OUTPUT:                                       │
│                                                             │
│      WITHOUT --standalone    │    WITH --standalone         │
│      ─────────────────────   │    ──────────────────        │
│      agent-browser commands  │    Playwright .js code       │
│                              │                              │
│      ```bash                 │    ```javascript             │
│      agent-browser open ...  │    const { chromium } = ...  │
│      agent-browser get ...   │    await page.goto(...)      │
│      agent-browser close     │    ```                       │
│      ```                     │                              │
└─────────────────────────────────────────────────────────────┘
```

## Tool Priority

| Operation | Primary Tool | Fallback | Notes |
|-----------|-------------|----------|-------|
| Find selectors for URL | `search_actions` | None | Search by domain/keywords |
| Get full selector details | `get_action_by_id` | None | Use action_id from search |
| List available sources | `list_sources` | `search_sources` | Browse all indexed sites |
| Generate agent-browser script | Agent (sonnet) | - | Default mode for /generate |
| Generate Playwright script | Agent (sonnet) | - | Use --standalone flag |
| Structure analysis | Agent (haiku) | - | Parse Actionbook response |
| Request new website | `agent-browser` | Manual | Submit to actionbook.dev (ONLY command that executes agent-browser) |

## Workflow Rules

### CRITICAL: Generate → Verify → Fix

**Every generated script MUST be verified by executing it.**

| Step | Action |
|------|--------|
| 1 | Generate script with Actionbook selectors |
| 2 | **Execute script to verify it works** |
| 3 | If failed: analyze error, fix script, go to step 2 |
| 4 | If success: output verified script + data preview |

### Verification Process

**For agent-browser scripts:**
```bash
# Execute each command
agent-browser open "https://example.com"
agent-browser wait --load networkidle
agent-browser get text ".selector"
# Check if data is returned
# If error → fix and retry
agent-browser close
```

**For Playwright scripts (--standalone):**
```bash
# Write to temp file and execute
node /tmp/scraper.js
# Check if output file has data
# If error → fix and retry
```

### Critical Rules

1. **ALWAYS verify generated scripts** - Execute and check BOTH parts
2. **Part 1: Script must run** - No errors, no timeouts
3. **Part 2: Data must be correct** - Not empty, not UI text, fields mapped correctly
4. **Fix errors automatically** - Don't output broken scripts or wrong data
5. **Use Actionbook MCP tools first** - Never guess selectors
6. **Include scroll handling** for lazy-loaded pages
7. **Include expand/collapse logic** for card-based layouts
8. **Always close browser** - Include `agent-browser close`
9.