create-scraper

What this skill does

# Create Scraper

Generate a complete, production-ready API documentation scraper. This workflow
supports Claude Code slash-command/subagent usage and Codex direct workflow
usage.

## What This Command Does

Given an API documentation URL, this command:

1. **Analyzes** the documentation site structure and patterns
2. **Designs** a scraper architecture tailored to that structure
3. **Generates** a complete Python scraper package
4. **Validates** the generated code works correctly

The result is a standalone scraper like `workato_scraper/` that can run forever without AI assistance.

## Workflow

Resolve the plugin root before loading bundled references:

- In Claude Code, use `${CLAUDE_PLUGIN_ROOT}`.
- In Codex from this repository, use `<repo-root>/scraper-generator`.
- In Codex from an installed plugin skill, resolve the plugin root as the
  directory two levels above this `SKILL.md`.

Codex does not load Claude plugin subagents or `Task` dispatch from this plugin.
Run the phases inline unless the user explicitly asks for Codex subagents or
parallel agent work.

### Phase 1: Site Analysis

First, understand the target documentation structure.

**Load the doc-site-analysis skill:**
Read `${PLUGIN_ROOT}/skills/doc-site-analysis/SKILL.md`

**Fetch and analyze the target URL in Claude Code:**
```
WebFetch {url}
```

**Fetch and analyze the target URL in Codex:**
Use an available repo-safe fetch method, such as `curl -L`, `python3` with
`urllib.request`, or Playwright only when static HTML is insufficient. Save a
small HTML sample under the chosen output directory when it helps make the
analysis repeatable.

Identify:
- Documentation framework (VuePress, Docusaurus, etc.)
- How endpoints are indexed (tables, sidebar, headings)
- How endpoint details are structured
- Where parameters, examples, descriptions are located
- Any edge cases or inconsistencies

**Document findings:**
Write analysis to `{output}/site-analysis.md`

Use the site-analyzer agent in Claude Code if the analysis is complex:
```
Task: Analyze the API documentation structure at {url}
Agent: site-analyzer
```

In Codex, perform this analysis inline by reading
`${PLUGIN_ROOT}/skills/doc-site-analysis/SKILL.md` and its targeted references.
Spawn a Codex subagent only when the user explicitly authorizes parallel agent
work.

### Phase 2: Architecture Design

Design the scraper based on discovered patterns.

**Load the scraper-architecture skill:**
Read `${PLUGIN_ROOT}/skills/scraper-architecture/SKILL.md`

**Study the reference implementation:**
Read `${PLUGIN_ROOT}/skills/scraper-architecture/examples/README.md`

Plan:
- Package structure
- Data models needed
- Extraction strategies
- Output formats

### Phase 3: Code Generation

Generate the complete scraper package.

**Load the code-generation skill:**
Read `${PLUGIN_ROOT}/skills/code-generation/SKILL.md`

**Generate all files:**

1. `{name}_scraper/__init__.py` - Package exports
2. `{name}_scraper/models.py` - Pydantic data models
3. `{name}_scraper/scraper.py` - HTTP fetching logic
4. `{name}_scraper/parser.py` - HTML parsing (the core logic)
5. `{name}_scraper/sections.py` - Section registry
6. `{name}_scraper/cli.py` - Command-line interface
7. `{name}_scraper/formatters/` - Output formatters
8. `main.py` - Entry point
9. `pyproject.toml` - Package configuration
10. `README.md` - Usage documentation

Use the code-generator agent in Claude Code for complex implementations:
```
Task: Generate scraper code based on site-analysis.md
Agent: code-generator
```

In Codex, generate the package inline using
`${PLUGIN_ROOT}/skills/code-generation/SKILL.md` and the architecture
references. Keep generated files inside the requested output directory.

### Phase 4: Validation

Verify the generated scraper works.

**Run the validation script:**
```bash
${PLUGIN_ROOT}/skills/code-generation/scripts/validate-scraper.sh {output}/{name}_scraper
```

Or use the validator agent in Claude Code:
```
Task: Validate the generated scraper at {output}/{name}_scraper
Agent: scraper-validator
```

In Codex, run the validation script directly from the resolved plugin root. If
the generated scraper has dependencies that are not installed, create an
isolated virtual environment inside the output directory or clearly report the
missing dependency as a validation blocker.

**Check:**
- Package imports without errors
- Scraper runs successfully
- Endpoints are extracted
- Output formats are valid

### Phase 5: Iteration (if needed)

If validation finds issues:

1. Read the validation report
2. Identify the root cause
3. Fix the issue in the generated code
4. Re-run validation
5. Repeat until passing

### Phase 6: Completion

When validation passes:

1. Summarize what was created
2. Provide usage instructions
3. Note any known limitations

## Example Session

**User input:**
```
/scraper-generator:create-scraper https://docs.workato.com/workato-api/connections.html
```

**Phase 1 output:**
```
Analyzing https://docs.workato.com/workato-api/connections.html...

Detected: VuePress framework
Index pattern: Quick reference table with Type/Resource/Description columns
Section pattern: H2 headings with explicit IDs
Found: 8 endpoints on this page

Wrote analysis to ./site-analysis.md
```

**Phase 2-3 output:**
```
Designing scraper architecture...
Using reference: workato_scraper

Generating package: workato_scraper/
  ✓ models.py (6 models)
  ✓ scraper.py (fetch with retry)
  ✓ parser.py (table + heading strategies)
  ✓ sections.py (28 sections)
  ✓ cli.py (argparse interface)
  ✓ formatters/ (JSON, Markdown, OpenAPI)
  ✓ main.py
  ✓ pyproject.toml
  ✓ README.md
```

**Phase 4 output:**
```
Validating generated scraper...
  ✓ Package structure valid
  ✓ Python syntax valid
  ✓ Package imports successfully
  ✓ Scraper runs (extracted 8 endpoints)
  ✓ JSON output valid
  ✓ OpenAPI spec valid

Validation: PASSED
```

**Phase 5 output:**
```
## Scraper Created Successfully!

Generated: workato_scraper/

### Usage:
```bash
# Install
uv venv && uv pip install -e .

# List sections
python main.py --list-sections

# Scrape connections
python main.py --section connections --verbose

# Scrape all sections
python main.py --all --verbose
```

### Output formats:
- JSON: Structured endpoint data
- Markdown: Human-readable documentation
- OpenAPI: Valid 3.0.3 specification

### Notes:
- 28 API sections configured
- 206+ endpoints across all sections
- All outputs validated successfully
```

## Key Principles

1. **Discovery-first** - Don't assume patterns; discover them
2. **Reference-informed** - Learn from workato_scraper example
3. **Validation-driven** - Test until it works
4. **Self-contained** - Generated code runs without AI

## Error Handling

**If site analysis fails:**
- Try alternative pages from the same site
- Look for API reference section
- Check if site requires authentication

**If code generation fails:**
- Simplify the extraction strategy
- Fall back to more generic patterns
- Generate skeleton with TODOs for complex parts

**If validation fails:**
- Read error messages carefully
- Fix one issue at a time
- Re-run validation after each fix