d4vinci/scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
How scrapling is put together
Scrapling is a Python web-scraping framework (v0.4.15) providing adaptive HTML parsing that relocates elements after site changes, multiple fetcher tiers (static requests via curl_cffi, stealth/dynamic headless browsers via Playwright/Patchright), and an asyncio spider framework with sessions, throttling, caching/checkpointing, robots.txt handling, and proxy rotation. Data flows from fetchers into Response objects wrapping the lxml-based parser, whose selections can be persisted to local storage for adaptive re-matching. A CLI exposes shell/MCP entry points ('scrapling' and 'scrapling-mcp'), and an agent-skill package plus Scrapy integration round out the distribution.
Languages
Frameworks
Infrastructure
Major components
scrapling.fetchers
Public API exposing Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher and their session variants over static and browser engines.
scrapling.engines.static
Sync/async HTTP session logic (_ASyncSessionLogic) built on curl_cffi for plain requests fetching.
scrapling.engines._browsers
Headless-browser controllers, page abstraction, stealth configuration, validators, and config tools for Playwright/Patchright-based fetching.
scrapling.parser
Adaptive HTML parsing/selection engine (CSS/XPath) that learns element signatures and relocates them when pages change.
scrapling.spiders
Async crawl framework: Spider base class, engine loop, SessionManager, scheduler, throttle, cache, checkpoints, link extraction, robots.txt compliance, and crawl templates (sitemap, feed, shopify, site_to_markdown).
scrapling.core
Shared primitives: adaptive-element storage, custom types, mixins, translator, AI helpers, interactive shell, and utility functions.
scrapling.engines.toolbelt
Fetch-time utilities: fingerprint/header generation, proxy rotation, ad/tracker domain blocking list, response conversion, navigation helpers.
scrapling.cli / integrations
CLI entry points including the MCP server ('scrapling-mcp') and a Scrapy integration adapter.
A quiet week with only a small documentation update.
5 weeksWeek of 2026-08-24
Week of 2026-08-24latest
A quiet week with only a small documentation update.
Week of 2026-08-17
A big release week (v0.4.15) centered on AI/RAG support, browser tab reuse, and MCP server improvements.
Week of 2026-08-10
Release v0.4.14 shipped with a key fix for finding elements by class name.
Week of 2026-08-03
MCP server upgraded to v2 with polished tooling, plus new XML/CSV export templates for spiders.
Week of 2026-07-27
A light maintenance week of typo fixes and CI adjustments.
Over the past month, Scrapling's development focused heavily on its AI and automation side: the MCP server was upgraded to v2 with new tools, sessions, and reliability fixes, while a new spiders feature made it easy to build RAG systems. Alongside that came several parser bug fixes around multi-class element matching, stealth improvements for bypassing Cloudflare challenges, and two releases (v0.4.14 and v0.4.15). The final week was quiet, with just a sponsor image update in the docs.
Week by week
2026-08-24A quiet week with only a small documentation update.latest1 change
Docs
Sponsor image update
Updated a sponsor image shown on the project's documentation pages.
2026-08-17A big release week (v0.4.15) centered on AI/RAG support, browser tab reuse, and MCP server improvements.6 changes
Feature
RAG system interface for spiders
Added an easy-to-use interface so users can build retrieval-augmented generation (AI) systems on top of scraped data.
Feature
Browser tabs stay open across requests
Fetchers can now keep browser tabs open and reuse them between requests, making repeated scraping faster.
Feature
MCP session tools
The MCP server gained tools to open request sessions and reuse them, plus a rename of 'get' to 'make_request' with more methods.
Fix
Stealth mode Cloudflare fixes
Cloudflare challenges are now solved regardless of language settings, with retry limits and no more mid-solve crashes.
Fix
Parser multi-class filtering fixed
Fixed cases where searching by CSS class silently missed elements with multiple classes, plus better handling of blank class filters.
Fix
Security and caching fixes
HTTP transport now requires authentication by default, cached responses keep their metadata, and low-retry requests fire exactly once.
2026-08-10Release v0.4.14 shipped with a key fix for finding elements by class name.4 changes
Fix
Multi-class element matching
Fixed find/find_all so they correctly match elements that have multiple CSS classes when filtering by class_.
Chore
Version bump to 0.4.14
Released version 0.4.14 with updated dependencies.
Docs
Stars history chart restored
Manually re-added the GitHub stars history image after GitHub API changes broke it.
Docs
Sponsor updates
Re-added one sponsor and added a new one to the documentation.
2026-08-03MCP server upgraded to v2 with polished tooling, plus new XML/CSV export templates for spiders.6 changes
Feature
XML/CSV feed templates
Spiders gained ready-made templates for exporting data as XML or CSV files, with documentation included.
Fix
MCP server upgrade to v2
Upgraded the MCP server to protocol v2 with shorter tool descriptions, server-level instructions, tool annotations, and metadata.
Feature
scrapling-mcp console shortcut
Added a simple command-line shortcut to register the MCP server more easily.
Fix
Browser version detection fix
Chrome version is now read directly from Playwright's package data instead of being guessed.
Refactor
Better MCP instructions
Added more specific guidance to the MCP server so AI assistants use it more effectively.
Docs
AI contribution policy and benchmarks
Published an AI policy for contributors and refreshed benchmark results with the latest numbers.
2026-07-27A light maintenance week of typo fixes and CI adjustments.2 changes
Docs
Documentation cleanups
Fixed several typos and small errors across the docs, and added a new sponsor.
Chore
CI pipeline fixes
Corrected issues in the automated testing/build pipelines.
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
No concrete code-level issues confirmed.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required