← All reportsPublic report

unclecode/crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

79k stars Python View on GitHubprofiled 11d ago
Try GitZoid
0PRs this week
0Contributors
126Deps scanned
12Issues found
01 · Repo overview

How crawl4ai is put together

Crawl4AI is an open-source, LLM-friendly web crawler and scraper written primarily in Python 3.10+. Its core is an asynchronous pipeline (AsyncWebCrawler) that drives headless Chromium/Firefox via Playwright and Patchright, scrapes HTML with lxml/BeautifulSoup strategies, applies pluggable extraction/chunking/markdown/content-filter strategies, and persists results through an async SQLite-backed cache layer. Configuration objects (BrowserConfig/CrawlerRunConfig) are serializable dataclass-style configs with an explicit trust-boundary layer that gates deserialization of untrusted request bodies on the Docker server. A secondary deployment surface under deploy/docker exposes the crawler as a FastAPI service with JWT-based auth, rate limiting, Prometheus metrics, Redis-backed work queues/caching, an MCP bridge, webhooks, and an egress proxy/broker. Releases are automated via split GitHub Actions workflows publishing to PyPI/GitHub Releases and multi-arch Docker images.

Languages

PythonJavaScriptYAMLDockerfile

Frameworks

PlaywrightPatchrightplaywright-stealthFastAPIuvicorn/gunicornlitellm (unclecode-litellm fork)BeautifulSoup4lxmlpydantichttpxaiohttpMCP SDK

Datastores

SQLite (via aiosqlite)Redis

Infrastructure

Docker / docker-composeGitHub Actions CI/CD (release.yml, docker-release.yml)GitHub Actions buildx cacheDocker Hub (unclecode/crawl4ai)PyPIsupervisord (deploy/docker/supervisord.conf)Prometheus metrics

Major components

Core crawler engine (async_webcrawler.py, async_crawler_strategy.py, browser_manager.py)

Orchestrates page fetching via managed Playwright/Patchright browser instances, including sessions, stealth, proxies, and page lifecycle.

Configuration & trust boundary (async_configs.py, config.py)

Defines BrowserConfig/CrawlerRunConfig and related configs, plus Provenance-gated deserialization allowlists that restrict what untrusted API bodies may construct.

Content scraping & markdown generation (content_scraping_strategy.py, markdown_generation_strategy.py, html2text/, content_filter_strategy.py)

Parses raw HTML into cleaned content, media/link inventories, metadata, and LLM-friendly Markdown using lxml/BeautifulSoup/html2text.

Extraction & chunking strategies (extraction_strategy.py, chunking_strategy.py, table_extraction.py)

Pluggable CSS/XPath/JSON/LLM-based structured-data extraction, regex chunking, and table extraction.

Deep crawling & URL discovery (deep_crawling/, domain_mapper.py, async_url_seeder.py, link_preview.py)

BFS/DFS/best-first recursive crawling with filters/scorers, plus domain-wide URL discovery from sitemaps, Common Crawl, Wayback, crt.sh, robots.txt, feeds, and probing.

Caching & persistence (async_database.py, cache_context.py, cache_validator.py)

Async SQLite-backed crawl-result caching with configurable CacheMode and freshness validation.

Docker serving layer (deploy/docker/: server.py, api.py, auth.py, auth_gate.py, schemas.py)

Exposes the crawler over HTTP/WebSocket/SSE with JWT authentication, rate limiting (slowapi), job handling, and Prometheus instrumentation.

Server infrastructure services (deploy/docker/: redis_config.py, work_queue.py, governor.py, crawler_pool.py, egress_proxy.py, egress_broker.py, llm_broker.py, mcp_bridge.py, webhook.py, monitor*.py)

Redis work queues, resource governance, browser pooling, network egress control, LLM key brokering, MCP protocol bridging, webhooks, and monitoring dashboards.

02 · Weekly changes

No digest this week

1 weeksWeek of

03 · Security check

Dependencies and code review

126 dependencies scanned

Dependency advisories

Security Watch

12 vulnerable packages (51 advisories) across 126 scanned dependencies.

criticalcritical · pypifixed in 1.84.0

litellm 1.74.0.post1

7 advisories. Fixed in 1.84.0.

How it can be abusedThese flaws let attackers slip past or abuse the login controls of the LiteLLM API proxy server itself, for example by forging host headers, reusing leaked password hashes, or promoting their own account to admin to mint keys and reach restricted routes. They only matter if you actually run the LiteLLM proxy as a service; a project like this one that uses LiteLLM merely as an optional client library for calling LLM providers during extraction does not expose those server endpoints.

  • criticalGHSA-4xpc-pv4p-pm3wLiteLLM: Authentication Bypass via Host Header Injection
  • highGHSA-53mr-6c8q-9789LiteLLM: Privilege escalation via unrestricted proxy configuration endpoint
  • highGHSA-69x8-hrgq-fjj8LiteLLM: Password hash exposure and pass-the-hash authentication bypass
  • highGHSA-7488-6r32-c95qLiteLLM: MCP Authentication Bypass via OAuth2 Passthrough Fallback
  • criticalGHSA-jjhc-v7c2-5hh6LiteLLM: Authentication bypass via OIDC userinfo cache key collision
  • highGHSA-qrc4-49gv-mv9mLiteLLM allows an authenticated internal_user to create API keys with access to routes that their role does not permit
  • highGHSA-wpfp-gwwc-vwq6LiteLLM allows a user to modify their own user_role via the /user/update endpoint
criticalcritical · pypifixed in 3.10.0

nltk 3.9.1

12 advisories. Fixed in 3.10.0.

How it can be abusedIf the crawler service ever passed user-controlled strings into NLTK's corpus loaders or download helpers, an attacker could craft crafted file paths to read arbitrary files from the server hosting Crawl4AI, such as its config files or JWT signing keys. Anyone running the Crawl4AI Docker API or MCP bridge that exposes these NLTK calls to untrusted requests would be affected.

  • highGHSA-469j-vmhf-r6v7NLTK has a Downloader Path Traversal Vulnerability (AFO) - Arbitrary File Overwrite
  • highGHSA-68j8-pq59-fqgmNLTK has a Path Traversal issue
  • highGHSA-6hm5-jgcp-p838Natural Language Toolkit (NLTK): Path Traversal in NKJPCorpusReader leads to Arbitrary File Read and bypasses the nltk.pathsec sandbox (ENFORCE=True)
  • criticalGHSA-7p94-766c-hgjpNLTK has a Zip Slip Vulnerability
  • highGHSA-848c-c2cx-j7qxNLTK vulnerable to Eval Injection via collocations CLI arguments
  • highGHSA-fg7f-2386-8897Natural Language Toolkit (NLTK): ReDoS in NLTK ReviewsCorpusReader FEATURES regex
  • highGHSA-h8wq-7xc4-p3qxNLTK has Arbitrary File Read via Absolute Path Input in nltk.util.filestring()
  • highGHSA-jm6w-m3j8-898gUnauthenticated remote shutdown in nltk.app.wordnet_app
  • highGHSA-m42h-3232-vpv3nltk: Arbitrary File Read via Path Traversal in nltk.data.load() through Percent-Encoded Sequences
  • highGHSA-p4gq-832x-fm9vNatural Language Toolkit (NLTK): URL-Encoded Path Traversal in nltk.data.load() Allows Arbitrary Local File Read
  • highGHSA-qvv7-cg9c-w4x3Natural Language Toolkit (NLTK): DNS-rebinding SSRF filter bypass in nltk.pathsec.urlopen (nltk.download / nltk.data.load) defeats ENFORCE mode
  • highGHSA-xh95-f55m-82fwNatural Language Toolkit (NLTK) has path traversal in FramenetCorpusReader.frame() that allows arbitrary XML file read, bypassing the nltk.pathsec sandbox (ENFORCE=True)
warninghigh · pypifixed in 3.13.3

aiohttp 3.12.13

2 advisories. Fixed in 3.13.3.

How it can be abusedAn attacker who controls or spoofs a page that Crawl4AI is asked to crawl can send a compressed response that expands enormously when unpacked, eating memory and stalling the crawler service for everyone using it. A second flaw lets a malformed chunked response from a hostile server crash or corrupt memory in the parsing code running inside your crawler process.

  • highGHSA-6mq8-rvhq-8wggAIOHTTP's HTTP Parser auto_decompress feature is vulnerable to zip bomb
  • highGHSA-cq5v-8q36-5273AIOHTTP: Out-of-bounds heap read in C HTTP response parser error path (malformed chunked response)
warninghigh · pypifixed in 1.2.0

brotli 1.1.0

1 advisory. Fixed in 1.2.0.

How it can be abusedCrawl4AI fetches pages from arbitrary URLs and automatically decompresses Brotli-compressed responses, so a malicious website it crawls can send specially crafted compressed data that makes the crawler burn memory or CPU and stall or crash, affecting anyone running crawls over untrusted sites, including the Docker API service that accepts crawl requests.

  • highGHSA-2qfp-q593-8484Scrapy is vulnerable to a denial of service (DoS) attack due to flaws in brotli decompression implementation
warninghigh · pypifixed in 48.0.1

cryptography 46.0.3

4 advisories. Fixed in 48.0.1.

How it can be abusedThe cryptography library ships its own copy of OpenSSL inside the installed package, so any Crawl4AI user crawling sites or running the Docker server makes HTTPS connections through that flawed crypto code; an attacker controlling a website being crawled could potentially exploit weaknesses in how connections are verified to intercept or tamper with data. Anyone deploying Crawl4AI as the FastAPI service or as a Python dependency is affected, since TLS handling happens automatically on every crawl.

  • highGHSA-537c-gmf6-5ccfVulnerable OpenSSL included in cryptography wheels
  • highGHSA-g6cj-pr64-35w5cryptography: PKCS#7 EnvelopedData decryption exposes a Bleichenbacher oracle through distinguishable errors and timing
  • highGHSA-jwv3-5hgf-82wwpython-cryptography: Duplicate self-signed intermediates can cause exponential path-building
  • highGHSA-r6ph-v2qm-q3c2cryptography Vulnerable to a Subgroup Attack Due to Missing Subgroup Validation for SECT Curves
warninghigh · pypifixed in 6.1.0

lxml 5.4.0

1 advisory. Fixed in 6.1.0.

How it can be abusedCrawl4AI fetches and parses whatever pages users point it at, so an attacker who controls a crawled URL can serve crafted XML that makes the parser read files from the machine running the crawler, such as config or credential files, and leak them into the crawl results shown back to the requester. Anyone running the crawler or its Docker service against untrusted URLs is affected.

  • highGHSA-vfmq-68hx-4jfwlxml: Default configuration of iterparse() and ETCompatXMLParser() allows XXE to local files
warninghigh · pypifixed in 12.3.0

pillow 10.4.0

13 advisories. Fixed in 12.3.0.

How it can be abusedAn attacker can point the crawler at a malicious page or submit a crafted file through the Docker API so that Pillow parses hostile image or font data, crashing the worker with huge decompressed images or corrupting memory. Anyone running the public-facing crawl service or crawling untrusted sites is affected.

  • highGHSA-45hq-cxwh-f6vcPillow `BdfFontFile`: `Image.new()` called without `_decompression_bomb_check()` — bomb protection bypass via font loading
  • highGHSA-5x94-69rx-g8h2Pillow: `FontFile.compile()`: `Image.new()` called without `_decompression_bomb_check()`
  • highGHSA-62p4-gmf7-7g93Pillow: Out-of-bounds read via attacker-controlled row stride on Pillow's mmap path (McIdas AREA files)
  • highGHSA-6r8x-57c9-28j4Pillow: Heap out-of-bounds write `Image.paste()` / `Image.crop()` via signed coordinate overflow
  • highGHSA-8v84-f9pq-wr9xPillow `PcfFontFile._load_bitmaps()`: `Image.frombytes()` called without `_decompression_bomb_check()` — bomb protection bypass via PCF font loading
  • highGHSA-9hw9-ch79-4vh6Pillow: Controlled heap out-of-bounds write in Pillow `ImageCmsTransform.apply()` via output mode mismatch
  • highGHSA-cfh3-3jmp-rvhcPillow affected by out-of-bounds write when loading PSD images
  • highGHSA-jjj6-mw9f-p565Pillow: Decompression Bomb DoS via PdfParser.PdfStream.decode()
  • highGHSA-phj9-mv4w-65pmPillow `GdImageFile._open()`: image dimensions accepted without `_decompression_bomb_check()`
  • highGHSA-pwv6-vv43-88grPillow has an OOB Write with Invalid PSD Tile Extents (Integer Overflow)
  • highGHSA-vjc4-5qp5-m44jPillow JPEG2000 tiled decode retains a growing scratch buffer and can be used for denial of service
  • highGHSA-whj4-6x5x-4v2jFITS GZIP decompression bomb in Pillow
  • highGHSA-xj96-63gp-2gmrPillow: Heap out-of-bounds write in `ImageFilter.RankFilter` via integer overflow in `ImagingExpand`
warninghigh · pypifixed in 26.0.0

pyopenssl 25.3.0

1 advisory. Fixed in 26.0.0.

How it can be abusedThis bug lives in a rarely used feature of pyOpenSSL for securing UDP-based connections, which this project never sets up; it only pulls the library in as an indirect dependency for normal HTTPS work. An attacker would need the crawler to be acting as a UDP TLS server with that specific handshake option enabled, so real users of Crawl4AI are effectively not exposed.

warninghigh · pypifixed in 6.14.1

pypdf 6.4.1

2 advisories. Fixed in 6.14.1.

How it can be abusedA crawler like this fetches whatever URL you point it at, including PDFs, so an attacker can host or link to a crafted PDF whose inline images never end. When the crawler tries to read that PDF, the parser spins forever inside the worker, tying up the crawler's capacity instead of returning results.

  • highGHSA-5xf7-4p34-54qrpypdf: Possible infinite loop for not terminated inline images
  • highGHSA-g867-7843-wf8qpypdf: Possible infinite loop for not terminated inline images (ASCII85 and ASCIIHex filter)
warninghigh · pypifixed in 2.8.4

soupsieve 2.7

2 advisories. Fixed in 2.8.4.

How it can be abusedCrawl4AI lets users pass CSS selectors in their crawl requests, and those selectors are fed to BeautifulSoup's matcher (soupsieve) against fetched pages. An attacker with access to the crawler service could submit a giant or pathologically crafted selector list to burn CPU or memory on the server, slowing or crashing crawling for everyone using that deployment.

  • highGHSA-2wc2-fm75-p42xSoup Sieve has Memory Exhaustion via Large Comma-Separated Selector Lists
  • highGHSA-836r-79rf-4m37Soup Sieve: Regular Expression Denial of Service (ReDoS) via Selector Parser
warninghigh · pypifixed in 5.3.0

transformers 4.53.1

2 advisories. Fixed in 5.3.0.

How it can be abusedIf this project ever loaded an image-matching model from an untrusted source, a crafted model file could run attacker code on the server during loading. However, Crawl4AI does not use the transformers library in its crawling pipeline, so real users of this repo are essentially unaffected.

  • highGHSA-29pf-2h5f-8g72HuggingFace transformers vulnerable to remote code execution
  • highGHSA-fgcw-684q-jj6rhuggingface/transformers: Arbitrary Code Execution During Model Initialization in the LightGlue Model Loading Path
warninghigh · pypifixed in 2.6.0

urllib3 2.5.0

4 advisories. Fixed in 2.6.0.

How it can be abusedA malicious website that Crawl4AI is asked to crawl can send a tiny response that expands into gigabytes when decompressed, or chain many redirects, exhausting memory and crashing the crawler. Anyone running the crawler, especially the Docker API service where users submit arbitrary URLs, is affected; a related flaw can also leak sensitive headers like auth tokens to a different server during redirects through a proxy.

  • highGHSA-2xpw-w6gg-jr37urllib3 streaming API improperly handles highly compressed data
  • highGHSA-38jv-5279-wg99Decompression-bomb safeguards bypassed when following HTTP redirects (streaming API)
  • highGHSA-gm62-xv2j-4w53urllib3 allows an unbounded number of links in the decompression chain
  • highGHSA-qccp-gfcp-xxvcurllib3: Sensitive headers forwarded across origins in proxied low-level redirects

Code review

21 files reviewed; 0 of 1 claimed findings confirmed after verification.

Get this report every week for your repos.

GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.

$19 a month, flat · First 10 outputs free · No card required