firecrawl/firecrawl
The context API to search, scrape, and interact with the web at scale. 🔥
How firecrawl is put together
Firecrawl is an open-source web scraping/crawling platform exposing an HTTP API to search, scrape, crawl, map, and interact with the web, returning LLM-ready Markdown or structured JSON. It is a pnpm monorepo whose core is a TypeScript/Express API (apps/api) backed by BullMQ queue workers, Redis, PostgreSQL/FoundationDB, and ClickHouse, complemented by supporting microservices: a Go HTML-to-Markdown converter (apps/go-html-to-md-service) and a Playwright-based browser rendering service (apps/playwright-service-ts). Requests flow from clients (many language SDKs) into the API, which enqueues scrape jobs to workers that fetch pages (via Playwright/fire-engine), convert HTML to Markdown, and optionally apply LLM extraction via the Vercel AI SDK across many providers. Data/results and billing are persisted to datastores, with Stripe/Autumn for billing and Sentry for observability.
Languages
Frameworks
Datastores
Infrastructure
Major components
apps/api
Core TypeScript/Express API server exposing scrape, search, crawl, map, agent, and interact endpoints.
Queue workers (apps/api/src/services)
BullMQ/nuq worker processes handling scraping, extraction, indexing, and log/concurrency jobs asynchronously.
scrapeURL engines (apps/api/src/scraper/scrapeURL)
Pluggable scraping engines including fire-engine that fetch and normalize page content.
apps/go-html-to-md-service
Standalone Go HTTP service that converts HTML to GitHub-flavored Markdown.
apps/playwright-service-ts
TypeScript browser-rendering microservice using Playwright and proxy-chain for JS-heavy pages.
apps/api/native
Rust/napi native module (koffi/wasi) providing performance-critical functionality to the API.
Language SDKs (apps/*-sdk)
Client libraries for Python, Node.js, Go, Rust, Java, .NET, Ruby, PHP, and Elixir.
Threat protection (apps/api/src/lib/threat-protection)
Shared provider/verdict type contract for request threat-protection logic.
Made the new billing rollout safer and improved search request tracing and caching.
5 weeksWeek of 2026-08-24
Week of 2026-08-24latest
Made the new billing rollout safer and improved search request tracing and caching.
Week of 2026-08-17
A heavy week of billing hardening, new rate-limit controls, and PDF page-attribution features.
Week of 2026-08-10
Rolled out the new billing routing with careful safeguards and stopped charging for failed requests.
Week of 2026-08-03
Expanded document parsing and PDF features while fixing billing accuracy and rate-limit regressions.
Week of 2026-07-27
Built and refined the new developer/code search product and modernized document format parsing.
Over the last several weeks the Firecrawl team steadily built out its new usage-based billing system ("firebill"), moving from an allowlist to a safe percentage-based rollout while tightening rules on what does and doesn't get charged. In parallel they launched and refined a new developer/code search product, replaced their document parser with a new engine, and kept up a constant stream of security patches and reliability fixes. The overall arc is one of monetization plumbing and search-product expansion, hardened by heavy testing and review.
Week by week
2026-08-24Made the new billing rollout safer and improved search request tracing and caching.latest3 changes
Feature
Gradual billing rollout by percentage
Switched the new billing system from a fixed allowlist to a sticky percentage-based ramp so orgs are added gradually and never bounced back and forth.
Feature
Request tracing through search
Passed a request ID through to the search engine so individual search requests can be tracked end to end.
Fix
Cache failed lookups
Started briefly remembering failed lookups and cleaned up how anonymous visitors are identified to reduce repeated wasted work.
2026-08-17A heavy week of billing hardening, new rate-limit controls, and PDF page-attribution features.6 changes
Feature
Per-organization rate-limit overrides
Added the ability to set custom request limits per customer and per endpoint, with a shortcut that skips extra lookups when an override applies.
Feature
PDF page attribution
Introduced page markers so extracted PDF text can be tied back to the specific page it came from.
Feature
Dedicated developer search
Rolled out a dedicated developer-focused search across the API and the JavaScript and Python SDKs.
Fix
Better research timeout handling
Detected and correctly reported research requests that time out while connecting, returning a proper gateway-timeout instead of a confusing error.
Chore
New default AI agent model
Added support for a newer agent model and adjusted defaults, gracefully handling unknown model names rather than failing.
Fix
Security audit fixes
Resolved flagged security vulnerabilities in third-party dependencies.
2026-08-10Rolled out the new billing routing with careful safeguards and stopped charging for failed requests.6 changes
Feature
Route billing through the new system
Began sending usage billing for approved organizations through the new firebill system with safeguards so charges aren't accidentally duplicated on retries.
Fix
Stop billing failed scrapes
Stopped charging customers for scrapes that fail because a website address can't be resolved.
Feature
Agent browser billing and views
Added billing for interactive agent browser sessions and exposed a way to see the active browser views.
Feature
Free research previews
Made research previews free by not recording preview credits against customer ledgers.
Fix
Branding response fixes
Improved brand detection so the correct brand name and colors are returned instead of generic page styling.
Fix
Restore image builds
Fixed the continuous-integration pipeline so container image builds run on GitHub again.
2026-08-03Expanded document parsing and PDF features while fixing billing accuracy and rate-limit regressions.6 changes
Feature
New document parser
Replaced the custom document parser with a new engine called anydoc for handling uploaded files.
Feature
PDF page-level output
Exposed per-page PDF text so results can show which page each section came from.
Fix
Restore dropped API routes
Brought back several labs and search routes that had been accidentally removed by an earlier rate-limit change.
Fix
Safe search option
Added a safe-search parameter to the newer search endpoint.
Fix
Correct text formatting
Stopped mangling plain-text output by no longer escaping certain characters like underscores.
Fix
Self-hosting reliability
Made database initialization repeatable so self-hosted setups don't break on restart.
2026-07-27Built and refined the new developer/code search product and modernized document format parsing.6 changes
Feature
Developer (code) search category
Introduced a new code-search capability, later renamed to developer search, with its own gateway route and gated behind a beta flag.
Feature
Search usage logging
Started logging developer-category searches to a dedicated ledger for tracking and billing.
Refactor
Rewritten legacy document providers
Rewrote the handling of older document formats like .doc, .docx, and .rtf to parse their formatting correctly.
Feature
Labs search proxy routes
Wired up several labs search endpoints so they proxy to the underlying labs service.
Fix
More reliable domain handling
Swapped in a better library for parsing website domains to correctly handle complex multi-part domain endings.
Fix
Billing timeout for history
Raised the timeout for historical usage lookups so large billing queries don't fail prematurely.
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
No concrete code-level issues confirmed.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required