extract

What this skill does

# stardust:extract

Crawl an existing website, parse each page, extract the brand surface,
and produce a stardust-formatted snapshot of the current state under
`stardust/current/`. The output describes what the site **is**; later
sub-commands consume it to decide what it **should be**.

This skill is **descriptive**: it does not invent direction, it does not
critique, and it does not modify the live site. It writes only under
`stardust/current/` and updates `stardust/state.json`.

## Inputs

- `<url>` — required. The origin to crawl. Examples: `https://example.com`,
  `https://example.com/shop`. A path narrows the same-origin crawl to
  that subtree.
- `--cap <N>` — optional. Override the default 5-page cap. The cap
  is intentionally small — a 5-page sample (home + four IA
  pillars/templates) is enough for cross-page brand aggregation,
  system-component detection, and the brand-review HTML to do
  useful work. Lift the cap with `--cap 25` (the previous default)
  or higher when a deeper crawl is genuinely needed.
- `--all` — optional. Lift the cap entirely; extract every
  discovered page after junk filtering. Equivalent to `--cap 0`.
  Use when the user spontaneously asks for a full crawl.
- `--pages <slug,slug,...>` — optional. Restrict the crawl to specific
  paths (slugs derived per `reference/ia-extraction.md`). Bypasses
  the cap.
- `--refresh <slug>` — optional. Re-extract one page that already exists
  in `state.json`.
- `--single` — optional. Equivalent to `--cap 1`. Useful for testing.
- `--wait <fast|medium|spec|auto>` — optional. Wait strategy per page.
  Default `medium`. See `reference/playwright-recipe.md` § Wait modes.
- `--no-junk-filter` — optional. Disable the default junk-page filter
  in discovery (see `reference/ia-extraction.md` § Filtering).
- `--no-consent-dismiss` — optional. Skip the pre-flight consent /
  cookie banner dismissal (see `reference/playwright-recipe.md`
  § Pre-flight: consent dismissal). Use when the redesign scope
  explicitly includes the consent surface, or when the
  dismissal's side-effects (script activation that wouldn't
  otherwise run) need to be avoided. Default behaviour is to
  dismiss; the contract preserves screenshots, voice
  aggregation, and per-section style from being polluted by
  the banner.
- `--prep` — optional. Run in **migrate-prep mode**: lift the cap,
  type each page, detect module candidates, capture typed content
  slots, emit the prep summary. See § Prep mode below. Typically
  invoked via the `prepare-migration` orchestrator skill rather
  than directly.

## Setup

Run the master skill's setup procedure first
(`skills/stardust/SKILL.md` § Setup): impeccable dep check, context
loader, state read.

Additional checks for this sub-command:

1. **Playwright availability.** The extraction step needs a real
   browser. Detect Playwright in this order: a Playwright MCP server,
   then `npx playwright`. If neither is available, stop and tell the
   user how to install Playwright.
2. **Origin collision.** If `stardust/state.json` already records
   `site.originUrl` and the new `<url>` is a different origin, stop and
   ask before clobbering. Stardust does not silently mix two sites in
   one project.
3. **Browser context.** Open a fresh `BrowserContext` for the run.
   Run the **consent dismissal pre-flight** per
   `reference/playwright-recipe.md` § Pre-flight: consent
   dismissal *unless* `--no-consent-dismiss` is set. Cookies
   persist across the per-page loop within the same context, so
   one dismissal covers the whole crawl. Record the resolved
   method in `_crawl-log.json#consent.method`.
4. **Bot-management probe.** When the first navigation in the run
   returns `ERR_HTTP2_PROTOCOL_ERROR` or `ERR_QUIC_PROTOCOL_ERROR`,
   or hangs through the entire hard-cap on what should be a fast
   origin, **do not retry headless**. Switch to
   `headless: false, channel: 'chrome'` per
   `reference/playwright-recipe.md` § Bot-management fallback and
   record the switch in `_crawl-log.json#discovery.fetchTechnique`
   so re-runs start in headed mode without rediscovering the
   issue.

## Procedure

### Phase 1 — Discovery

Discover the page inventory before crawling. Procedure in
`reference/ia-extraction.md`. In summary:

1. Fetch `<origin>/sitemap.xml`, then `<origin>/sitemap_index.xml`,
   then check `robots.txt` for `Sitemap:` directives.
2. If no sitemap is reachable, run a same-origin BFS crawl from
   `<url>`, depth-limited to 3, link-extracting from rendered HTML.
3. Filter the discovered URL list: same origin only, exclude
   `mailto:`, `tel:`, anchor-only links, query-only variations,
   common asset paths (`.css`, `.js`, `.pdf`, image extensions).
4. De-duplicate trailing-slash variations.
5. Apply the junk-page filter (`reference/ia-extraction.md` §
   Junk-page filter) unless `--no-junk-filter` is set. Surface the
   filtered list to the user as overridable.
6. Apply the cap (default 5, or `--cap`, or `--all` for no cap)
   and **proceed silently**. Print an informational summary of
   what was kept and what was cut — but do **not** gate on user
   confirmation. The default cap is small enough that the common
   case is "extract 5 pages and move on"; pausing for a yes/no
   reply on every run is friction without value. Users who want
   different scope set it spontaneously at command time:

   ```
   $stardust extract https://example.com              # default 5 pages
   $stardust extract https://example.com --cap 25     # bump to 25
   $stardust extract https://example.com --all        # lift the cap
   $stardust extract https://example.com --pages home,about,pricing
   $stardust extract https://example.com --single     # just the entry URL
   ```

   The agent reads spontaneous scope intent from the user's prompt
   (e.g. "extract all pages", "look at just the home and pricing",
   "do a full crawl") and applies the equivalent flag. No
   re-confirmation needed once intent is clear.

   Informational output (not a prompt — proceed immediately):

   ```
   Discovered 38 pages on https://example.com (sitemap.xml).
   Filtered as likely junk (5): /test/, /sample-page/, /holiday1/, ...
   Selecting 5 highest-priority pages:
     - / (home)
     - /about
     - /pricing
     - /products
     - /contact

   Cut (28 pages, --all to lift): /blog/post-1, /blog/post-2, ...

   Extracting...
   ```

   Selection heuristic: page-type checklist first, then score-based
   ranking (home + IA-pillar keywords + sitemap priority − archive /
   version markers). See `reference/ia-extraction.md` § Page
   selection and § Priority for the cap. The English-only keyword
   list is a known limitation for localized sites.

7. Write the discovered list to `stardust/current/_crawl-log.json`
   (created if absent) with `_provenance` and the full discovery
   reasoning, including `filteredAsJunk[]` and `userChoice`. This is
   an audit trail, not a state file.

### Phase 2 — Per-page extraction

For each page in the cap-respecting list, render with Playwright
following `reference/playwright-recipe.md`. The recipe is mandatory —
in particular, do not skip the wait, scroll, or capture-list steps:

- Viewport 1440 × 900 @ 2× DPR
- Wait per the configured wait mode (default `medium`; see § Wait
  modes in `reference/playwright-recipe.md`)
- Disable animations via `prefers-reduced-motion: reduce`
- After the wait resolves, scroll to bottom in 4 viewport-height
  steps with 300 ms pauses, then return to top — this is required
  to trigger lazy-load and IntersectionObserver-driven content
- Record `waitMs` and `waitMode` in the per-page `_provenance`

Capture per page (full schema in `reference/current-state-schema.md`):

- Page metadata (title, meta description, OG tags, theme-color)
- Semantic structure: heading outline, landmark roles, sections
- Content: visible text per section (full innerText, **no
  truncation** per `reference/playwright-recipe.md` § Capture
  list 7), structured paragraphs (`bod