firecrawl
Official Firecrawl CLI skill for web scraping, search, crawling, and browser automation. Returns clean LLM-optimized markdown. USE FOR: - Web search and research - Scraping pages, docs, and articles - Site mapping and bulk content extraction - Browser automation for interactive pages Must be pre-installed and authenticated. See rules/install.md for setup, rules/security.md for output handling.
What this skill does
# Firecrawl CLI Web scraping, search, and browser automation CLI. Returns clean markdown optimized for LLM context windows. Run `firecrawl --help` or `firecrawl <command> --help` for full option details. ## Prerequisites Must be installed and authenticated. Check with `firecrawl --status`. ``` 🔥 firecrawl cli v1.8.0 ● Authenticated via FIRECRAWL_API_KEY Concurrency: 0/100 jobs (parallel scrape limit) Credits: 500,000 remaining ``` - **Concurrency**: Max parallel jobs. Run parallel operations up to this limit. - **Credits**: Remaining API credits. Each scrape/crawl consumes credits. If not ready, see [rules/install.md](rules/install.md). For output handling guidelines, see [rules/security.md](rules/security.md). ```bash firecrawl search "query" --scrape --limit 3 ``` ## Workflow Follow this escalation pattern: 1. **Search** - No specific URL yet. Find pages, answer questions, discover sources. 2. **Scrape** - Have a URL. Extract its content directly. 3. **Map + Scrape** - Large site or need a specific subpage. Use `map --search` to find the right URL, then scrape it. 4. **Crawl** - Need bulk content from an entire site section (e.g., all /docs/). 5. **Browser** - Scrape failed because content is behind interaction (pagination, modals, form submissions, multi-step navigation). | Need | Command | When | | --------------------------- | --------- | --------------------------------------------------------- | | Find pages on a topic | `search` | No specific URL yet | | Get a page's content | `scrape` | Have a URL, page is static or JS-rendered | | Find URLs within a site | `map` | Need to locate a specific subpage | | Bulk extract a site section | `crawl` | Need many pages (e.g., all /docs/) | | AI-powered data extraction | `agent` | Need structured data from complex sites | | Interact with a page | `browser` | Content requires clicks, form fills, pagination, or login | See also: [`download`](#download) -- a convenience command that combines `map` + `scrape` to save an entire site to local files. **Scrape vs browser:** - Use `scrape` first. It handles static pages and JS-rendered SPAs. - Use `browser` when you need to interact with a page, such as clicking buttons, filling out forms, navigating through a complex site, infinite scroll, or when scrape fails to grab all the content you need. - Never use browser for web searches - use `search` instead. **Avoid redundant fetches:** - `search --scrape` already fetches full page content. Don't re-scrape those URLs. - Check `.firecrawl/` for existing data before fetching again. **Example: fetching API docs from a large site** ``` search "site:docs.example.com authentication API" → found the docs domain map https://docs.example.com --search "auth" → found /docs/api/authentication scrape https://docs.example.com/docs/api/auth... → got the content ``` **Example: data behind pagination** ``` scrape https://example.com/products → only shows first 10 items, no next-page links browser "open https://example.com/products" → open in browser browser "snapshot -i" → find the pagination button browser "click @e12" → click "Next Page" browser "scrape" -o .firecrawl/products-p2.md → extract page 2 content ``` **Example: login then scrape authenticated content** ``` browser launch-session --profile my-app → create a named profile browser "open https://app.example.com/login" → navigate to login browser "snapshot -i" → find form fields browser "fill @e3 '[email protected]'" → fill email browser "click @e7" → click Login browser "wait 2" → wait for redirect browser close → disconnect, state persisted browser launch-session --profile my-app → reconnect, cookies intact browser "open https://app.example.com/dashboard" → already logged in browser "scrape" -o .firecrawl/dashboard.md → extract authenticated content browser close ``` **Example: research task** ``` search "firecrawl vs competitors 2024" --scrape -o .firecrawl/search-comparison-scraped.json → full content already fetched for each result grep -n "pricing\|features" .firecrawl/search-comparison-scraped.json head -200 .firecrawl/search-comparison-scraped.json → read and process what you have → notice a relevant URL in the content scrape https://newsite.com/comparison -o .firecrawl/newsite-comparison.md → only scrape this new URL ``` ## Output & Organization Unless the user specifies to return in context, write results to `.firecrawl/` with `-o`. Add `.firecrawl/` to `.gitignore`. Always quote URLs - shell interprets `?` and `&` as special characters. ```bash firecrawl search "react hooks" -o .firecrawl/search-react-hooks.json --json firecrawl scrape "<url>" -o .firecrawl/page.md ``` Naming conventions: ``` .firecrawl/search-{query}.json .firecrawl/search-{query}-scraped.json .firecrawl/{site}-{path}.md ``` Never read entire output files at once. Use `grep`, `head`, or incremental reads: ```bash wc -l .firecrawl/file.md && head -50 .firecrawl/file.md grep -n "keyword" .firecrawl/file.md ``` Single format outputs raw content. Multiple formats (e.g., `--format markdown,links`) output JSON. ## Commands ### search Web search with optional content scraping. Run `firecrawl search --help` for all options. ```bash # Basic search firecrawl search "your query" -o .firecrawl/result.json --json # Search and scrape full page content from results firecrawl search "your query" --scrape -o .firecrawl/scraped.json --json # News from the past day firecrawl search "your query" --sources news --tbs qdr:d -o .firecrawl/news.json --json ``` Options: `--limit <n>`, `--sources <web,images,news>`, `--categories <github,research,pdf>`, `--tbs <qdr:h|d|w|m|y>`, `--location`, `--country <code>`, `--scrape`, `--scrape-formats`, `-o` ### scrape Scrape one or more URLs. Multiple URLs are scraped concurrently and each result is saved to `.firecrawl/`. Run `firecrawl scrape --help` for all options. ```bash # Basic markdown extraction firecrawl scrape "<url>" -o .firecrawl/page.md # Main content only, no nav/footer firecrawl scrape "<url>" --only-main-content -o .firecrawl/page.md # Wait for JS to render, then scrape firecrawl scrape "<url>" --wait-for 3000 -o .firecrawl/page.md # Multiple URLs (each saved to .firecrawl/) firecrawl scrape https://firecrawl.dev https://firecrawl.dev/blog https://docs.firecrawl.dev # Get markdown and links together firecrawl scrape "<url>" --format markdown,links -o .firecrawl/page.json ``` Options: `-f <markdown,html,rawHtml,links,screenshot,json>`, `-H`, `--only-main-content`, `--wait-for <ms>`, `--include-tags`, `--exclude-tags`, `-o` ### map Discover URLs on a site. Run `firecrawl map --help` for all options. ```bash # Find a specific page on a large site firecrawl map "<url>" --search "authentication" -o .firecrawl/filtered.txt # Get all URLs firecrawl map "<url>" --limit 500 --json -o .firecrawl/urls.json ``` Options: `--limit <n>`, `--search <query>`, `--sitemap <include|skip|only>`, `--include-subdomains`, `--json`, `-o` ### crawl Bulk extract from a website. Run `firecrawl crawl --help` for all options. ```bash # Crawl a docs section firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json # Full crawl with depth limit firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json # Che
Related in AI Agents
skill-development
IncludedComprehensive meta-skill for creating, managing, validating, auditing, and distributing Claude Code skills and slash commands (unified in v2.1.3+). Provides skill templates, creation workflows, validation patterns, audit checklists, naming conventions, YAML frontmatter guidance, progressive disclosure examples, and best practices lookup. Use when creating new skills, validating existing skills, auditing skill quality, understanding skill architecture, needing skill templates, learning about YAML frontmatter requirements, progressive disclosure patterns, tool restrictions (allowed-tools), skill composition, skill naming conventions, troubleshooting skill activation issues, creating custom slash commands, configuring command frontmatter, using command arguments ($ARGUMENTS, $1, $2), bash execution in commands, file references in commands, command namespacing, plugin commands, MCP slash commands, Skill tool configuration, or deciding between skills vs slash commands. Delegates to docs-management skill for official documentation.
reprompter
IncludedTransform messy prompts into well-structured, effective prompts — single or multi-agent. Use when: "reprompt", "reprompt this", "clean up this prompt", "structure my prompt", rough text needing XML tags and best practices, "reprompter teams", "repromptception", "run with quality", "smart run", "smart agents", multi-agent tasks, audits, parallel work, anything going to agent teams. Don't use when: simple Q&A, pure chat, immediate execution-only tasks. See "Don't Use When" section for details. Outputs: Structured XML/Markdown prompt, quality score (before/after), optional team brief + per-agent sub-prompts, agent team output files. Success criteria: Single mode quality score ≥ 7/10; Repromptception per-agent prompt quality score 8+/10; all required sections present, actionable and specific.
adaptive-compaction
IncludedAdaptive add-on policy and recovery layer that decides WHEN to compact, prune, snapshot, or fork -- replacing fixed-percent auto-compaction across Claude Code, Codex, and MCP-capable hosts. Trigger on auto-compact timing or damage: "when should I compact", "is it safe to compact now or start a fresh session", "auto-compact fires too early/mid-task", "switching to an unrelated task but the window still has space", "context rot", "answers get worse the longer the session runs", "the agent forgot the plan or my decisions after it summarized", "add a layer on top that manages context without changing the agent", raising autoCompactWindow to give the policy room, or installing/tuning a cross-tool compaction policy or PreCompact hook -- even when "compaction" is never said but the problem is context-window pressure or post-summarization memory loss. Do NOT use to summarize a conversation, build RAG, write a summarization prompt (decides WHEN not HOW), or answer max-context-length trivia.
agent-skill-creator
IncludedCreate cross-platform agent skills from workflow descriptions. Activates when users ask to create an agent, automate a repetitive workflow, create a custom skill, or need advanced agent creation. Triggers on phrases like create agent for, automate workflow, create skill for, every day I have to, daily I need to, turn process into agent, need to automate, create a cross-platform skill, validate this skill, export this skill, migrate this skill. Supports single skills, multi-agent suites, transcript processing, template-based creation, interactive configuration, cross-platform export, and spec validation.
llm-wiki
IncludedUse when building or maintaining a persistent personal knowledge base (second brain) in Obsidian where an LLM incrementally ingests sources, updates entity/concept pages, maintains cross-references, and keeps a synthesis current. Triggers include "second brain", "Obsidian wiki", "personal knowledge management", "ingest this paper/article/book", "build a research wiki", "compound knowledge", "Memex", or whenever the user wants knowledge to accumulate across sessions instead of being re-derived by RAG on every query.
skill-master
IncludedAgent Skills authoring, evaluation, and optimization. Create, edit, validate, benchmark, and improve skills following the agentskills.io specification. Use when designing SKILL.md files, structuring skill folders (references, scripts, assets), ingesting external documentation into skills, running trigger evals, benchmarking skill quality, optimizing descriptions, or performing blind A/B comparisons. Keywords: agentskills.io, SKILL.md, skill authoring, eval, benchmark, trigger optimization.