scraper-studio

What this skill does

# Bright Data — Scraper Studio

Build a scraper from natural language, then run it. Two commands live in this skill:

- **`bdata scraper create <url> <description>`** — describe what you want in plain English; Bright Data's AI Flow generates a scraper template and returns a `collector_id`.
- **`bdata scraper run <collector_id> <url>`** — run that collector (or any existing one from the Bright Data web UI) against a URL and get the extracted data back.

**The bridge between the two is `collector_id`.** It is printed by `create` and consumed by `run`. Always save it.

For pre-built scrapers on platforms like Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc., **stop and use the [`data-feeds`](../data-feeds/SKILL.md) skill instead** — those scrapers already exist, are faster, cheaper, and more reliable than building a new one. Use Scraper Studio when no pre-built scraper covers the target site, or when the user wants a custom shape of output for an existing platform.

## Setup gate (run first)

```bash
if ! command -v bdata >/dev/null 2>&1; then
    echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md"
elif ! bdata zones >/dev/null 2>&1; then
    echo "bdata not authenticated — run: bdata login  (or: bdata login --device for SSH)"
fi
```

Halt and route to setup if either check fails. Both commands require an authenticated CLI.

## Pick your path

| Situation | Action |
|---|---|
| User describes data they want from a URL, no scraper exists yet | `bdata scraper create <url> "<description>"` → save the `collector_id` |
| User has a `collector_id` and wants data from one URL | `bdata scraper run <collector_id> <url>` (default async + poll) |
| User has a `collector_id` and wants data from many URLs | `bdata scraper run <collector_id> --urls "u1,u2,..."` or `--input-file urls.txt` (single batch call) |
| Page is small and you want fast feedback (≤ ~50 s, single URL) | `bdata scraper run … --sync` |
| Scraper ran but returned wrong / empty / partial data | inspect the output, then `bdata scraper heal <collector_id> "<what's wrong>"` → review preview → approve → re-run to verify |
| Site is a known platform (Amazon, LinkedIn, TikTok, …) | **stop — use `data-feeds` skill** |
| You want SERP / discovery, not extraction | **use `search` skill** |
| You want a one-off raw page fetch | **use `scrape` skill** |

---

## Action 1 — `scraper create`

Generate a scraper from a URL + plain-English description.

```bash
bdata scraper create <url> "<description>" [--name <name>] \
    [--deliver-webhook <url>] [--timeout <seconds>] \
    [--json | --pretty] [-o <path>] [--timing] [-k <api-key>]
```

The description is the most important input. A good description names every field you want and any conditions on how to find them. See [references/prompts.md](references/prompts.md) for examples of strong vs. weak descriptions.

```bash
# Minimal
bdata scraper create https://example.com/product/1 \
    "Extract title, price, currency, image URL, and availability \
     from this product page. If the price has a strike-through \
     original price, capture both as price and original_price."

# Save the full AI output for inspection
bdata scraper create https://example.com/product/1 \
    "Extract title, price, and image URL" \
    --name product-scraper-v1 \
    --pretty -o create.json
```

### What happens under the hood

`create` chains three Bright Data API calls — surface this to the user so they can debug from logs:

1. **`POST /dca/collector`** — creates an empty scraper template with a stub webhook delivery target (`https://example.com/webhook` by default). Returns a `collector_id` like `c_mp3tuab31lswoxvpws`.
2. **`POST /dca/collectors/{collector_id}/automate_template`** — triggers Bright Data's AI Flow with the description + URL.
3. **`GET .../automate_template/progress`** (polled) — waits for `status: "done"`. Generation typically takes **5–10 minutes** for moderately complex pages.

### Critical: hold the `collector_id`

Every failure path in `create` (AI trigger fails, polling times out, generation finishes with `status: "failed"`) **still leaves a partially-built collector** at the printed `collector_id`. Always tell the user the id is recoverable — they can:

- Open `https://brightdata.com/cp/scrapers/{collector_id}` to finish or inspect it in the web UI.
- Re-trigger generation programmatically against the same id.
- Delete it from the UI if they want a clean slate.

Never claim "create failed, start over" without surfacing the `collector_id` from the response.

### `--timeout` — default 600 s

AI generation can run 5–10 min for complex pages. If the page is simple, the default is plenty. For an elaborate description on a heavy site, raise it:

```bash
bdata scraper create https://complex-site.com/page \
    "Extract all 30 fields …" \
    --timeout 1200
```

On `Timeout after N seconds waiting for AI generation`, the `collector_id` is still printed. Re-check progress in the web UI rather than re-running `create` (which builds a new collector).

### `--deliver-webhook` — placeholder by default

The CLI sets a stub webhook (`https://example.com/webhook`). This satisfies the API contract but **does not deliver anything**. For CLI use, leave the stub — you'll fetch results synchronously via `scraper run`. The real delivery target can be reconfigured in the [Bright Data web UI](https://brightdata.com/cp/scrapers) if the user wants webhook delivery for production runs.

---

## Action 2 — `scraper run`

Execute a scraper against one or more URLs.

```bash
bdata scraper run <collector_id> [url] \
    [--urls "u1,u2,..." | --input-file <path>] \
    [--sync [--sync-timeout 25-50]] \
    [--timeout <seconds>] [--name <name>] [--version <version>] \
    [--json | --pretty] [-o <path>] [--timing] [-k <api-key>]
```

**Pick exactly one input source:** positional `<url>`, `--urls`, or `--input-file`. Combining them errors with `only one input source`.

Multi-URL routes through `/dca/trigger` as a single API call with an array body — the canonical pattern from the [Scraper Studio Node](https://github.com/brightdata/bright-data-scraper-studio-nodejs-project) and [Python](https://github.com/brightdata/bright-data-scraper-studio-python-project) reference projects (`triggerWithUrls` / `trigger_with_urls`). One snapshot, one poll loop, one merged result array. **Do not hand-roll a `for url in $(cat urls.txt); do bdata scraper run ...` loop** — that's N API calls for what should be one.

### Choosing the run mode

```
                ┌─────────────────────────────────────────────────┐
                │ Expected to finish in ≤ ~50 seconds?            │
                └─────────────────────────────────────────────────┘
                 yes                                       no
                  │                                         │
                  ▼                                         ▼
       ┌──────────────────────┐                ┌────────────────────────┐
       │   --sync             │                │  default (no flag)     │
       │   /dca/crawl         │                │  trigger_immediate +   │
       │   one-shot, 25–50 s  │                │  poll get_result       │
       └──────────────────────┘                └────────────────────────┘
                  │
                  │  on 202 timeout: response_id is printed —
                  │  re-run WITHOUT --sync to poll for it
                  ▼
            (cleanly fall back to async)
```

| Mode | Endpoint | When to use |
|---|---|---|
| **Default (single URL, async + poll)** | `/dca/trigger_immediate` → poll `/dca/get_result` | Anything you expect to take more than ~50 s, anything paginated, anything you're not sure about. This is the safe default for one URL. |
| **`--sync` (single URL)** | `/dca/crawl` | Single-page extractions you expect to complete in under 50 s. Faster path: one request, no polling. **Incompatible with multi-URL** — `/dca/crawl` accept