scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
How scrapy is put together
Scrapy is a Python web crawling and scraping framework (Python 3.10+) built on the Twisted asynchronous networking engine, with growing asyncio integration (pytest runs with --reactor=asyncio). Its architecture is a pipeline of cooperating components: a Crawler orchestrates an ExecutionEngine that pulls Requests from a Scheduler, sends them through downloader middlewares to downloaders, passes Responses through spider middlewares into Spiders, and feeds scraped Items through item pipelines and exporters. Behavior is configured via a layered Settings system, extensible middleware/extension managers, and a CLI (scrapy.cmdline) exposing commands like crawl, runspider, shell, and startproject. Data flows from start requests through dupefilters/prioritized queues, HTTP(S)/HTTP2/FTP/data-URI downloads, parsing via parsel/lxml selectors, to feed exports (local, S3, GCS optional).
Languages
Frameworks
Infrastructure
Major components
scrapy.core.engine/scheduler/scraper
The execution engine core that coordinates request scheduling, concurrent downloading, and response scraping.
scrapy.downloadermiddlewares
Downloader-side hooks for cookies, redirects, retries, compression, proxy, HTTP auth, robots.txt, offsite filtering, caching, and stats.
scrapy.spidermiddlewares
Spider-output processing chain for depth, URL length, referer policy, HTTP error handling, start-request handling, and meta copying.
scrapy.http
Request/Response abstractions including headers, cookie jars, and responsetypes detection.
scrapy.extensions
Lifecycle extensions such as feed export, HTTP cache, telnet console, remote control, memory/stats monitoring, throttling, and log stats.
scrapy.pipelines
Item pipelines for media (files/images) downloading and processing.
scrapy.commands + cmdline
CLI entry point implementing crawl, runspider, shell, fetch, parse, genspider, startproject, settings, bench, and other commands.
scrapy.utils
Broad utility layer covering defer/asyncio helpers, reactor management, compression, SSL, FTP, sitemap, logging, serialization, and download-handler wiring.
A quieter week focused on CI reliability, test cleanup, and typing work following the 2.18 release.
5 weeksWeek of 2026-08-24
Week of 2026-08-24latest
A quieter week focused on CI reliability, test cleanup, and typing work following the 2.18 release.
Week of 2026-08-17
Scrapy 2.18.0 was released alongside a remote control extension, Python 3.15 support, and a large batch of beginner-friendly documentation.
Week of 2026-08-10
A feature-rich week adding FTPS exports, ISO 8601 date serialization, HTTP/2 improvements, and clearer failure messages.
Week of 2026-08-03
A large week of polish: brotli became a required dependency, spiders gained runtime flexibility, and docs expanded across cookies, optimization, and item exporters.
Week of 2026-07-27
Infrastructure and quality took center stage: CI was rebuilt around uv and CodSpeed benchmarks, while features like Request.to_curl() and unlimited concurrency landed.
Over the past five weeks, Scrapy shipped version 2.18.0 and pushed hard on both user-facing features (a remote control extension, FTPS feed exports, better cookie handling, browser-like bad header handling) and a large wave of documentation aimed at beginners. In parallel, the team invested heavily in engineering health: faster and more reliable CI, extensive type annotations, flaky-test fixes, performance benchmarks, and broader test coverage.
Week by week
2026-08-24A quieter week focused on CI reliability, test cleanup, and typing work following the 2.18 release.latest5 changes
Chore
CI coverage patch split across job groups
The code-coverage measurement setup was divided across different CI job groups so no single job is overloaded.
Refactor
Default settings file fully typed
The file holding all default settings now has complete type annotations, letting the team drop a special exception in their type-checking configuration.
Chore
CI jobs given explicit time limits
Every continuous-integration job that lacked one now has a timeout so hung runs fail instead of stalling forever.
Chore
Tests re-enabled after reactor-error comments
Tests that had been skipped due to a known Twisted shutdown warning were turned back on.
Chore
Test parallelism tuned
The test runner was switched to spawn one worker per CPU core for more efficient use of CI machines.
2026-08-17Scrapy 2.18.0 was released alongside a remote control extension, Python 3.15 support, and a large batch of beginner-friendly documentation.6 changes
Feature
Remote control extension added
A new extension lets you control a running crawl remotely instead of only through the command line.
Docs
Beginner's guide to main concepts
A new tutorial page walks newcomers through Scrapy's core ideas from scratch.
Feature
Spider templates can live anywhere
The project-template option now accepts a direct path to a template file, not just built-in names.
Feature
open_in_browser handles non-text responses
The open_in_browser helper now respects the response's Content-Type, so images and other binary content open correctly.
Chore
Python 3.15 support in CI
Continuous integration now tests against the upcoming Python 3.15, with related test expectations updated.
Fix
Numerous flaky tests stabilized
Several intermittently failing tests (browser exploit checks, page-count limits, Windows timing) were made reliable.
2026-08-10A feature-rich week adding FTPS exports, ISO 8601 date serialization, HTTP/2 improvements, and clearer failure messages.6 changes
Feature
FTPS support for FTP feed exports
Exported data can now be sent to FTP servers over encrypted FTPS connections.
Feature
Dates serialized as ISO 8601
The built-in JSON encoder now writes dates and times in the standard ISO format instead of ad-hoc strings.
Feature
HTTP/2 handler promoted out of experimental
The Twisted-based HTTP/2 downloader is no longer labeled experimental, though its API became private; it also gained new frame-size tuning and signal support.
Fix
Browser-like bad header handling
The default download handler now deals with malformed HTTP headers the way web browsers do, avoiding crashes.
Fix
Clear error when Spider.start() is misused
Spiders whose start method isn't an async generator now fail with an understandable message rather than confusing errors.
Fix
Better exit codes on startup failures
The check, parse, and fetch commands now return a proper error exit code when they can't start.
2026-08-03A large week of polish: brotli became a required dependency, spiders gained runtime flexibility, and docs expanded across cookies, optimization, and item exporters.6 changes
Feature
Brotli compression always available
Brotli decompression moved from optional extra to a built-in dependency, so compressed sites work out of the box.
Feature
Spiders can change allowed_domains at run time
A spider may now adjust which domains it's allowed to crawl while running, plus a new depth_reset metadata key controls link-depth tracking.
Fix
Cookie lookup fixed for dotless hosts and IPs
Cookies now work correctly for hostnames without dots (like localhost) and raw IP addresses.
Docs
New cookies and optimization doc pages
Two new documentation pages explain how cookies are handled and how to make crawls faster.
Refactor
Link canonicalization sped up
Each extracted link is now normalized once instead of repeatedly, cutting redundant work on link-heavy pages.
Fix
GCS exports need only object-level permissions
Google Cloud Storage feed exports were adjusted so service accounts with limited, object-level access can be used.
2026-07-27Infrastructure and quality took center stage: CI was rebuilt around uv and CodSpeed benchmarks, while features like Request.to_curl() and unlimited concurrency landed.6 changes
Feature
Request.to_curl() added
Any request can now be converted into a ready-to-run curl command, handy for debugging outside Scrapy.
Feature
Unlimited concurrency via CONCURRENT_REQUESTS = 0
Setting concurrency to zero now means 'no limit' rather than being invalid.
Feature
Robots.txt crawl delays exposed
A new crawl_delay() method and robots_parsed signal let spiders respect per-site delay rules more easily.
Chore
CI modernized with uv and CodSpeed
Continuous integration now installs dependencies with the fast uv tool, runs fewer but smarter jobs, and includes CodSpeed performance benchmarking.
Fix
Feed export batches delivered promptly
Fixed a bug where full export batches weren't handed off immediately, delaying output.
Fix
Case-insensitive dict copy bug fixed
Copying a case-insensitive dictionary no longer accidentally shares internal state with the original.
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
No concrete code-level issues confirmed.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required