What it does
The infrastructure for “clean, LLM-ready data” from the live web is a real bottleneck for AI agents and RAG pipelines. General scrapers leave you to handle JavaScript rendering, complex markup, robots.txt, and multi-step interactions yourself — and the output rarely lands in a shape that an LLM can consume directly.
Firecrawl bundles that infrastructure into one API. Quoting firecrawl.dev: “the infrastructure layer that helps AI find, read, and act on the live web.” Output is returned as LLM-ready markdown or structured data from the start.
Key features — three integrated capabilities
-
Search — web search
Run a query and get search results, with optional content extraction for each hit in the same call.
-
Scrape — page → clean data
Extract a single URL into JSON, markdown, or branding formats. JavaScript rendering and complex markup are handled automatically.
-
Interact — page automation
Automate clicks, typing, and navigation to reach content that static scraping cannot.
Additional endpoints include Agent (autonomous multi-source research), Crawl (multi-page extraction with depth and page limits), Map (discover indexed URLs on a site), and Batch Scrape (parallel processing of many URLs).
Cloud vs Open Source
| Aspect | Open Source (this repo) | Cloud (firecrawl.dev) |
|---|---|---|
| Operator | You | Firecrawl team |
| License | AGPL-3.0 (SDKs / some UI = MIT) | SaaS terms |
| Extra features | Core engine | Additional cloud-only features (see README comparison) |
| Cost | Your infra cost | 1,000 credits/month free + paid plans |
| Data control | Full self-control | Routed through Firecrawl infrastructure |
| Best for | Strict data residency, cost or customization control | Fast start without infrastructure overhead |
SDKs
| Language | Install |
|---|---|
| Python | pip install firecrawl-py |
| Node.js | npm install @mendable/firecrawl-js |
| Java | JitPack via Gradle / Maven (com.github.firecrawl:firecrawl-java-sdk:2.0) |
| Elixir | {:firecrawl, "~> 1.0"} |
| Rust | firecrawl = "2" |
A community Go SDK is linked separately in the README.
Usage
Cloud (fastest start) — generate an API key at firecrawl.dev and call directly.
curl -X POST 'https://api.firecrawl.dev/v2/search' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"query": "firecrawl", "limit": 5}'Self-host — use the docker-compose stack at the repo root.
git clone https://github.com/firecrawl/firecrawl
cd firecrawl
docker compose upSee SELF_HOST.md in the repo for environment setup and dependencies.
From Claude Code — use the Firecrawl MCP. Point it at a self-hosted instance via FIRECRAWL_API_URL to keep the cloud out of the loop entirely.
Notes
- AGPL-3.0 has real obligations — review the copyleft terms before integrating the engine source into a commercial product. Simply calling the API as a client (via MCP or SDK) is generally unaffected.
- SDKs and some UI components are MIT — explicit in the README. Client-side integration draws only the MIT-licensed parts.
- robots.txt respected by default — README quote: “Firecrawl respects robots.txt by default,” and: “It is the sole responsibility of end users to respect websites’ policies when scraping.”
- Adoption — firecrawl.dev cites over one million signups and customers including Apple, Canva, and Lovable.
- Actively maintained — near-daily commits since the first commit in April 2024.