web-crawler

What this skill does

## Preferred entry: call exports.py (don't hand-roll requests)
Ready-made helpers live in `skills/web-crawler/exports.py`. Prefer them over
writing your own `proxied_get`/`proxied_post` calls — they already inject the
proxy credentials, so **there is no API key to find** (don't read
`$SCRAPECREATORS_API_KEY` / `$FIRECRAWL_API_KEY`, don't check `.env`, don't ask
the user).

```python
import sys; sys.path.insert(0, "/data/workspace/skills/web-crawler")
from exports import scrape_markdown, youtube_transcript, sc_get
scrape_markdown("https://example.com/article")          # Firecrawl fallback
youtube_transcript("https://youtube.com/watch?v=ID")     # ScrapeCreators
sc_get("/v1/tiktok/profile", handle="charlidamelio")     # any SC endpoint

from exports import archive_fallback                      # paywall / Firecrawl-403
archive_fallback("https://www.nytimes.com/.../article.html")  # archive snapshot
```

Named wrappers exist for the high-frequency actions (YouTube/TikTok transcript &
video, IG/Twitter/Reddit posts, profiles, Google/Reddit search). For any other
ScrapeCreators endpoint use `sc_get(path, **params)` — it auto-strips leading
`@`/`#` from handles/hashtags. The intent-routing tables below still tell you
*which* endpoint to pass. Pass `caller_id="chat:<thread>"` (or `job:`/`preview:`)
for cost tracking.

## Quick trigger rules (read this first)
Use this skill immediately when any of these conditions is true:

- `web_fetch` returns HTTP **401/403/429/5xx**
- Response is an anti-bot challenge page (for example Cloudflare "Attention Required", "Just a moment", or challenge/captcha pages)
- The page is JS-heavy and the first fetch misses required detail fields (for example publish time, author, listing code, updated time, price breakdown)
- Search results do not contain the requested field and the value must be extracted from the target page itself

Fallback rule:
- If ordinary fetch is blocked or incomplete, switch to Firecrawl fallback in this skill before asking the user for screenshots/manual text.

## Error signatures -> action
| Signature | Action |
|---|---|
| `web_fetch` HTTP 401/403/429/5xx | Call Firecrawl `POST /v2/scrape` once with `formats:["markdown","links"]` + `onlyMainContent:true` |
| Cloudflare/challenge page text in body | Same Firecrawl call as above |
| Markdown still misses key fields | Retry once with `formats:["rawHtml"]` |
| **Firecrawl itself returns 403 / empty** (hard paywall: NYT, WSJ, Economist, FT, Bloomberg) | Call `archive_fallback(url)` — recovers full text from a web archive snapshot |

### Paywall / Firecrawl-blocked fallback chain (use `archive_fallback`)
When Firecrawl can't get the page either (it returns 403, or markdown comes back
empty) the site is behind a hard paywall or aggressive WAF. Do NOT keep retrying
Firecrawl. Recover the article from a **web archive snapshot** instead:

```python
from exports import archive_fallback
res = archive_fallback("https://www.nytimes.com/.../article.html")
res["markdown"]      # full text, or "" if no snapshot exists anywhere
res["source"]        # "archive.today" | "wayback" | None
res["snapshot_url"]  # the snapshot that was scraped
```

How it works (and why this order):
- **archive.today first** (`archive.ph` / `archive.is` mirrors). User-triggered,
  real-browser captures; historically preserves full text *behind* paywalls.
  Best bet for NYT/WSJ/Economist. We scrape its `/newest/` snapshot via Firecrawl
  (archive.today has its own Cloudflare, so scrape it through Firecrawl, never
  `web_fetch` it directly).
- **Wayback Machine second** (`archive.org`). Automated crawler that *honors*
  robots.txt and paywalls, so it often has NO full text for hard paywalls — but
  it's a good fallback for ordinary 403/Cloudflare pages that aren't paywalled.

Limitation: archives only return text **someone already saved**. If
`res["markdown"] == ""`, no snapshot exists — stop, tell the user, and try the
outlet's official API/RSS or a different source. Do not fabricate the article.

## What each service is for
### ScrapeCreators — Social media data extraction (27+ platforms)

Use for any request involving social media profiles, posts, videos, comments, transcripts, search, ads, trending content, or engagement metrics. Covers TikTok, Instagram, YouTube, LinkedIn, Facebook, Twitter/X, Reddit, Threads, Bluesky, Pinterest, Snapchat, Twitch, Kick, Truth Social, TikTok Shop, Google search, and link-in-bio services (Linktree, Komi, Pillar, Linkbio, Linkme, Amazon Shop).

**Base URL:** `https://api.scrapecreators.com`
**Auth:** No user-supplied key needed. sc-proxy injects platform credentials automatically — just send the request. The `x-api-key` header can be any value or omitted entirely. Do NOT bail out or ask the user for a key if `$SCRAPECREATORS_API_KEY` looks unset; that env var is intentionally not required.
**Method:** All endpoints use GET requests with query params. Responses are JSON.

### Firecrawl — Fallback web page scraper

Only a fallback crawler for one web page when ordinary fetching fails. Use `POST /v2/scrape` with a single `url` and focused formats like `markdown`, `html`, `rawHtml`, `links`, `summary`, or constrained `json`/`question`/`highlights` extraction.

**Auth:** No user-supplied key needed. sc-proxy injects the Firecrawl credential automatically when you call through `core.http_client.proxied_post` — just send the request. Do NOT read `$FIRECRAWL_API_KEY` from env, do NOT check `.env` for it, and do NOT ask the user for a Firecrawl key if it looks unset; that env var is intentionally not required. The same proxy-injection model as ScrapeCreators applies here.

Do not use Firecrawl crawl/map/search/agent/browser endpoints. Do not request screenshots, audio, branding, images, or browser actions unless the proxy policy is expanded later.

---

## ScrapeCreators — Intent routing
Map user intent to the right endpoint. Endpoint paths use the pattern `/v1/platform/action`.

**Important:** After selecting an endpoint from the tables below, fetch its OpenAPI spec at `https://docs.scrapecreators.com/{path}/openapi.json` for full parameter details, types, and example response before making the actual API call. For example: `https://docs.scrapecreators.com/v1/tiktok/profile/openapi.json`

### Profiles / User Info
| Platform | Endpoint | Primary Param | Example |
|----------|----------|---------------|---------|
| TikTok | `/v1/tiktok/profile` | handle | `stoolpresidente` |
| Instagram | `/v1/instagram/profile` | handle | `jane` |
| YouTube | `/v1/youtube/channel` | handle, channelId, or url | `ThePatMcAfeeShow` |
| LinkedIn (person) | `/v1/linkedin/profile` | url | `https://www.linkedin.com/in/parrsam/` |
| LinkedIn (company) | `/v1/linkedin/company` | url | `https://linkedin.com/company/shopify` |
| Facebook | `/v1/facebook/profile` | url | `https://www.facebook.com/mantraindianfolsom` |
| Twitter/X | `/v1/twitter/profile` | handle | `elonmusk` |
| Reddit | `/v1/reddit/subreddit/details` | subreddit or url | `AskReddit` |
| Threads | `/v1/threads/profile` | handle | `zuck` |
| Bluesky | `/v1/bluesky/profile` | handle | `jay.bsky.team` |
| Pinterest | `/v1/pinterest/user/boards` | handle | `pinterest` |
| Truth Social | `/v1/truthsocial/profile` | handle | `realDonaldTrump` |
| Twitch | `/v1/twitch/profile` | handle | `ninja` |
| Snapchat | `/v1/snapchat/profile` | handle | `djkhaled` |

### Posts / Content Feeds
| Platform | Endpoint | Primary Param | Example |
|----------|----------|---------------|---------|
| TikTok videos | `/v3/tiktok/profile/videos` | handle | `stoolpresidente` |
| Instagram posts | `/v2/instagram/user/posts` | handle | `jane` |
| Instagram reels | `/v1/instagram/user/reels` | handle or user_id | `jane` or `2700692569` |
| Instagram highlights | `/v1/instagram/user/highlights` | handle or user_id | `jane` or `2700692569` |
| YouTube videos | `/v1/youtube/channel/videos` | handle or channelId | `ThePatMcAfeeShow` |
| YouTube shorts | `/v1/you