mrscraper
Run AI-powered, unblockable web scraping, data extraction with natural language via the MrScraper API
What this skill does
# MrScraper Run AI-powered, unblockable web scraping, data extraction with natural language via the MrScraper API ## Actions This skill supports: - Opening blocked pages through unblocker (stealth browser + IP rotation) - Starting AI scraper runs from natural-language instructions - Rerunning existing scraper configurations on one or multiple URLs - Running manual workflow-based reruns - Fetching paginated results and detailed results by ID This skill is API-only and does not depend on bundled local scripts. ## Base URLs - Unblocker API: `https://api.mrscraper.com` - Platform API: `https://api.app.mrscraper.com` ## Authentication ### Unblocker API auth Use query-param auth on unblocker endpoint: - `token=<MRSCRAPER_API_TOKEN>` ### Platform API auth Use header-based auth on platform endpoints: ```http x-api-token: <MRSCRAPER_API_TOKEN> accept: application/json content-type: application/json ``` ### How to get `MRSCRAPER_API_TOKEN`? An API token lets your applications securely interact with MrScraper APIs and rerun scrapers created in the dashboard. Follow these steps in the dashboard: 1. Click your **User Profile** at the top-right corner. 2. Select **API Tokens**. 3. Click **New Token**. 4. Enter a **name** and set an **expiration date**. 5. Click **Create**. 6. Copy the new token and store it securely as `MRSCRAPER_API_TOKEN`. 7. Use it in requests through the `x-api-token` header. Security rule: - Never expose tokens in client-side code (browser/mobile app bundles). - Store tokens in environment variables or server-side secret managers. Notes from the auth docs: - The API key works for all V3 Platform endpoints. - The same key can be used for endpoints on `sync.scraper.mrscraper.com`. - For access to endpoints on other hosts, contact `[email protected]`. ## Install and Runtime - No local install step is required by this skill document. - No bundled `scripts/` are required. - Calls are direct HTTPS requests to the two base URLs above. ## Data and Scope - Data is sent only to `api.app.mrscraper.com` and `api.mrscraper.com`. - Responses may contain extracted page content and scrape metadata. - This skill does not define hidden persistence or background jobs. - Never expose tokens in logs, commits, or output. ## Endpoints ### 1. Unblocker - Method: `GET` - URL: `https://api.mrscraper.com` - Auth: `token` query parameter Opens a target URL through stealth browsing and IP rotation, then returns HTML. Use this when direct access is blocked by captcha or anti-bot protections. #### Query parameters: | Field | Type | Required | Default | Description | | ---------------- | --------- | -------- | ------- | --------------------------------------- | | `token` | `string` | Yes | — | Unblocker token (`MRSCRAPER_API_TOKEN`) | | `url` | `string` | Yes | — | URL-encoded target URL | | `timeout` | `number` | No | 60 | Max wait in seconds (example `120`) | | `geoCode` | `string` | No | None | Geographic routing code (example `SG`) | | `blockResources` | `boolean` | No | false | Block non-essential resources | #### Request example: ```bash curl --location 'https://api.mrscraper.com?token=<MRSCRAPER_API_TOKEN>&timeout=120&geoCode=SG&url=https%3A%2F%2Fwww.lazada.sg%2Fproducts%2Fpdp-i111650098-s23209659764.html&blockResources=false' ``` #### Response example: ```html <!doctype html> <html> <head>...</head> <body>...</body> </html> ``` #### Notes: - Prefer explicit `geoCode` and practical timeouts for repeatable behavior. - Only pass cookies when session-specific content is required. ### 2. Create AI Scraper - Method: `POST` - Host: `https://api.app.mrscraper.com` - Path: `/api/v1/scrapers-ai` - Auth: `x-api-token` Create a new AI scraper run from natural-language instructions. #### Payload parameters (for `agent`: `general` or `agent`: `listing`): | Field | Type | Required | Default | Description | | -------------- | -------- | -------- | -------- | ---------------------------------------------------------- | | `url` | string | Yes | — | Target URL | | `message` | string | Yes | — | Extraction instruction | | `agent` | string | No | general | The AI agent type to use for scraping: `general`, `listing`, or `map` | | `proxyCountry` | string | No | None | ISO country code for proxy-based scraping | #### Payload parameters (for `agent`: `map`): | Field | Type | Required | Default | Description | | ----------------- | -------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------- | | `url` | `string` | Yes | — | Target URL | | `agent` | `string` | No | map | The AI agent type to use for scraping (for this case it is `map`) | | `maxDepth` | `number` | No | 2 | Maximum depth level for crawling links from the starting URL.<br>0 = only the starting URL, 1 = +direct links | | `maxPages` | `number` | No | 50 | Maximum number of pages to scrape during the crawling process. | | `limit` | `number` | No | 1000 | Maximum number of data records to extract across all pages. Scraping stops when this limit is reached. | | `includePatterns` | `string` | No | "" | Regex patterns to include (separate multiple with `\|\|`) | | `excludePatterns` | `string` | No | "" | Regex patterns to exclude (separate multiple with `\|\|`) | #### Request example: ```bash curl -X POST "https://api.app.mrscraper.com/api/v1/scrapers-ai" \ -H "x-api-token: <MRSCRAPER_API_TOKEN>" \ -H "Content-Type: application/json" \ -d '{ "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "message": "Extract title, price, stocks, and rating", "agent": "general" }' ``` #### Response example: ```json { "id": "497f6eca-6276-4993-bfeb-53cbbbba6f08", "createdAt": "2019-08-24T14:15:22Z", "createdById": "e13e432a-5323-4484-a91d-b5969bc564d9", "updatedAt": "2019-08-24T14:15:22Z", "updatedById": "d8bc6076-4141-4a88-80b9-0eb31643066f", "deletedAt": "2019-08-24T14:15:22Z", "deletedById": "8ef578ad-7f1e-4656-b48b-b1b4a9aaa1cb", "userId": "2c4a230c-5085-4924-a3e1-25fb4fc5965b", "scraperId": "6695bf87-aaa6-46b0-b1ee-88586b222b0b", "type": "AI", "url": "http://example.com", "status": "Finished", "error": "string", "tokenUsage": 0, "runtime": 0, "data": {}, // MAIN SCRAPED DATA "htmlPath": "string", "recordingPath": "string", "screenshotPath": "string", "dataPath": "string" } ``` #### Notes: - Choose agent type correctly as each agent is specialized for specified use cases. Use `general` for most standard web scraping tasks. The go to agent if the user doesn't specify or the connected LLM is not confident about the type of page. But mostly used for scraping product page, but handles any type of page very well as well. Use `listing` for scraping listing pages like product listings, job listings, etc. Choose this if the connected LLM can confidently identify whether the given URL is a listing page. Use `map` for crawling and getting all subdomain or subpages of a website. Choose this if the user specifies that the given URL is a website and not a specific page. For `map` agent type, there is a special arg
Related in Backend & APIs
jfrog
IncludedInteract with the JFrog Platform via the JFrog CLI and REST/GraphQL APIs. Use this skill when the user wants to manage Artifactory repositories, upload or download artifacts, manage builds, configure permissions, manage users and groups, work with access tokens, configure JFrog CLI servers, search artifacts, manage properties, set up replication, manage JFrog Projects, run security audits or scans, look up CVE details, query exposures scan results from JFrog Advanced Security, manage release bundles and lifecycle operations, aggregate or export platform data, or perform any JFrog Platform administration task. Also use when the user mentions jf, jfrog, artifactory, xray, distribution, evidence, apptrust, onemodel, graphql, workers, mission control, curation, advanced security, exposures, or any JFrog product name.
cupynumeric-migration-readiness
IncludedPre-migration readiness assessor for porting NumPy to cuPyNumeric. Use BEFORE substantial porting work begins when the user asks whether code will scale on GPU, whether they should migrate to cuPyNumeric, which NumPy patterns transfer cleanly, what must be refactored before porting, or mentions pre-port assessment, scaling analysis, or refactor planning. Inspect the user's source code, look up NumPy usage, cross-reference the cuPyNumeric API support manifest, and distinguish distributed-scaling-friendly patterns from blockers such as unsupported APIs, scalar synchronization, host round-trips, Python/object-heavy control flow, shape/data-dependent branching, and in-place mutation hazards. Produce a verdict of READY, LIGHT REFACTOR, SIGNIFICANT REFACTOR, or NOT RECOMMENDED, with concrete refactor pointers.
alibabacloud-data-agent-skill
IncludedInvoke Alibaba Cloud Apsara Data Agent for Analytics via CLI to perform natural language-driven data analysis on enterprise databases. Data Agent for Analytics is an intelligent data analysis agent developed by Alibaba Cloud Database team for enterprise users. It automatically completes requirement analysis, data understanding, analysis insights, and report generation based on natural language descriptions. This tool supports: discovering data resources (instances/databases/tables) managed in DMS, initiating query or deep analysis sessions, real-time progress tracking, and retrieving analysis conclusions and generated reports. Use this Skill when users need to query databases, analyze data trends, generate data reports, ask questions in natural language, or mention "Data Agent", "data analysis", "database query", "SQL analysis", "data insights".
token-optimizer
IncludedReduce OpenClaw token usage and API costs through smart model routing, heartbeat optimization, budget tracking, and native 2026.2.15 features (session pruning, bootstrap size limits, cache TTL alignment). Use when token costs are high, API rate limits are being hit, or hosting multiple agents at scale. The 4 executable scripts (context_optimizer, model_router, heartbeat_optimizer, token_tracker) are local-only — no network requests, no subprocess calls, no system modifications. Reference files (PROVIDERS.md, config-patches.json) document optional multi-provider strategies that require external API keys and network access if you choose to use them. See SECURITY.md for full breakdown.
resend-cli
IncludedUse this skill when the task is specifically about operating Resend from an AI agent, terminal session, or CI job via the official resend CLI: installing/authenticating the CLI, sending/listing/updating/cancelling emails, batch sends, domains and DNS, webhooks and local listeners, inbound receiving, contacts, topics, segments, broadcasts, templates, API keys, profiles, or debugging Resend CLI/API failures. Trigger on mentions of Resend CLI, `resend`, `resend doctor`, `resend emails send`, `resend domains`, `resend webhooks listen`, `resend emails receiving`, or agent-friendly terminal automation.
alibabacloud-odps-maxframe-coding
IncludedUse this skill for MaxFrame SDK development and documentation navigation on Alibaba Cloud MaxCompute (ODPS). Helps answer MaxFrame API, concept, official example, and supported pandas API questions; create data processing programs; read/write MaxCompute tables; debug jobs (remote or local); and build custom DPE runtime images. Trigger when users mention MaxFrame, MaxCompute with MaxFrame, ODPS table processing, DPE runtime, MaxFrame docs/examples, DataFrame/Tensor operations, or GPU runtime setup. Works for both English and Chinese queries about Alibaba Cloud data processing with MaxFrame.