← All reportsPublic report

nanmicoder/mediacrawler

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 | 评论爬虫、微博帖子 | 评论爬虫、百度贴吧帖子 | 百度贴吧评论回复爬虫 | 知乎问答文章|评论爬虫

64k stars Python View on GitHubprofiled 11d ago
Try GitZoid
17PRs this week
4Contributors
389Deps scanned
12Issues found
01 · Repo overview

How mediacrawler is put together

MediaCrawler is a multi-platform social-media scraping tool (Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, Zhihu) written primarily in Python 3.11+. It logs into platforms via Playwright browser automation or Chrome DevTools Protocol (CDP) against a local Chrome instance, preserving login state and evaluating JS snippets instead of reverse-engineering signature algorithms; platform-specific JS signing scripts (libs/douyin.js, libs/zhihu.js) are executed via pyexecjs/Node.js. Data flow: main.py parses CLI args, CrawlerFactory dispatches to the selected platform crawler, whose client fetches notes/comments, results are normalized into model objects and persisted by the store layer to JSONL/JSON/CSV/Excel/SQLite/MySQL/PostgreSQL/MongoDB. A secondary FastAPI backend (api/) plus a React/Vite/Tailwind WebUI (webui/) lets users configure, launch, monitor (WebSocket), and export crawl runs.

Languages

PythonTypeScriptJavaScriptSQL

Frameworks

FastAPIPlaywrightReactViteTailwind CSSRadix UIZustandTanStack QueryVitePressTyper

Datastores

SQLiteMySQLPostgreSQLMongoDBRedis

Infrastructure

Chrome DevTools Protocol (CDP) mode against local Chrome >= 144 on 127.0.0.1:9222Node.js >= 16 runtime for JS signature executionuvicorn API server (default port 8080)Vite dev server (port 5173) proxying /api to 8080

Major components

CLI entry & crawler orchestration (main.py, cmd_arg/, var.py)

Parses command-line arguments, initializes databases, instantiates the right platform crawler via the CrawlerFactory registry, and handles graceful shutdown/wordcloud/Excel flush.

media_platform/* (xhs, dy, ks, bili, wb, tieba, zhihu packages)

One self-contained crawler package per platform, each with client.py (API calls), core.py (crawl flow), login.py (qrcode/cookie login), help.py (signature helpers), field.py, exception.py.

config/

Python-module configuration: global crawl behavior (platform, keywords, login type, CDP mode, concurrency) in base_config.py, per-platform knobs in <platform>_config.py, DB connection settings in db_config.py.

api/ (FastAPI backend)

Serves the WebUI, exposes REST endpoints for crawler control, data preview/export, environment checks, and a WebSocket router for live status/logs; manages crawler subprocesses via services/crawler_manager.py.

webui/ (React SPA)

Vite-built TypeScript front end (React 18, Tailwind, Radix UI, Zustand, i18next) for configuring and monitoring crawls, communicating over axios REST calls and WebSocket hooks.

store/<platform>/

Per-platform persistence adapters (_store_impl.py plus *_store_media.py) that write crawled notes/comments to the configured sink; shared ExcelStoreBase handles buffered Excel output.

database/

Async DB access layer: session management, SQLAlchemy models, Alembic migrations, auto table creation, and mongodb_store_base for document storage.

proxy/ and tools/

IP-proxy pool abstraction with provider plugins (kuaidaili, wandouhttp, jishu_http) and utilities for CDP browser launching, HTTP clients (httpx_util), sliding-captcha handling, file writers, and wordcloud generation.

Over the past few weeks, the MediaCrawler project focused on making its data collection more reliable across platforms. Fixes targeted error handling and retry behavior on Xiaohongshu (Little Red Book) and Bilibili, rate-limit resilience on Kuaishou, and a timezone bug on Weibo that skewed timestamps by eight hours. Documentation updates rounded out the work.

Week by week

2026-08-10A busy week centered on fixing Xiaohongshu and Bilibili collection errors so the crawler stops failing or re-fetching unnecessarily when a site restricts access.latest6 changes

Fix

Fixed access-restriction crashes on Xiaohongshu and Bilibili

The crawler now handles cases where these sites block or limit access without breaking the collection run.

Fix

Fixed comment collection edge cases on Xiaohongshu and Bilibili

Comment gathering no longer misses items at collection boundaries.

Fix

Better classification of raw Xiaohongshu responses

Error responses from Xiaohongshu are now categorized correctly so the tool doesn't repeatedly retry requests it shouldn't.

Chore

Added regression tests for Xiaohongshu responses and retries

New automated tests make sure the response handling and retry fixes keep working over time.

Chore

Merged community contributions

Two external fix proposals were reviewed and merged into the main codebase.

Docs

Updated the README

Project documentation was refreshed with several small updates.

2026-08-03Kuaishou got the most attention this week, with new request signing support and smarter handling of server rate limits, plus a fix for missing pinned comments on Bilibili.5 changes

Fix

Missing pinned comments recovered on Bilibili

Pinned (highlighted) comments at the top of video discussions are now captured correctly.

Feature

Kuaishou now supports signed web requests

Requests to Kuaishou's web API are properly signed, letting the crawler talk to the site successfully.

Fix

Reduced Kuaishou rate-limit problems with retry backoff

When Kuaishou throttles the crawler, it now waits and retries instead of hammering the server.

Refactor

Cleaner terminal logs for Kuaishou

Console output during Kuaishou runs is now condensed into readable summaries per item.

Docs

Replaced broken star-history charts in README

Documentation images that had stopped rendering were swapped out for working ones.

2026-07-27A small but impactful week: a Weibo bug that made all post timestamps appear eight hours late was fixed.2 changes

Fix

Weibo timestamps no longer off by 8 hours

Converting Weibo's date format previously dropped the timezone offset, shifting every timestamp; it now preserves the correct time.

Chore

Merged the timezone fix from a contributor

The community-submitted patch was reviewed and merged into the main branch.

2026-07-20Maintenance-focused week: an internal signing library for Xiaohongshu was upgraded and some temporary workaround code removed, alongside a documentation refresh.3 changes

Fix

Upgraded the xhshow signing library to 0.2.0

The component that signs Xiaohongshu requests was updated to a newer version.

Refactor

Removed temporary GET-signing workaround

A stopgap code patch used for GET request signatures is no longer needed thanks to the library upgrade.

Docs

README updated

Project documentation received another round of updates.

03 · Security check

Dependencies and code review

389 dependencies scanned

Dependency advisories

Security Watch

12 vulnerable packages (26 advisories, 1 actively exploited) across 389 scanned dependencies.

warninghigh · npmfixed in 6.2.4

vite 5.4.6

3 advisories, actively exploited. Fixed in 6.2.4.

How it can be abusedIf a developer runs the Vite dev server for the webui while it is reachable from other machines, an attacker could read files outside the project folder through crafted requests (including Windows path tricks) or trigger code execution through the editor launcher on Windows. This affects developers running the dev tools, not end users of the crawler.

  • mediumGHSA-4r4m-qw57-chr8exploitedVite has a `server.fs.deny` bypassed for `inline` and `raw` with `?import` query
  • highGHSA-c27g-q93r-2cwflaunch-editor vulnerable to command injection via the crafted request on Windows
  • highGHSA-fx2h-pf6j-xcffvite: `server.fs.deny` bypass on Windows alternate paths
criticalcritical · pypi

asyncmy 0.2.10

1 advisory.

How it can be abusedMediaCrawler can save crawled notes and comments to MySQL through asyncmy, and this flaw lets SQL be smuggled in through dictionary keys used when building queries. Anyone running the crawler against hostile or manipulated platform content and writing results to MySQL could have extra commands run inside their database, though in this project the record keys come from fixed internal models rather than attacker input, so exploiting it would require unusual setup.

warninghigh · pypifixed in 0.3.0

aiomysql 0.2.0

1 advisory. Fixed in 0.3.0.

How it can be abusedIf you configure MediaCrawler to save crawl results to a MySQL database that someone else controls or has compromised, that fake or malicious database server can trick your machine into sending back any file it asks for, such as your .env file with database passwords and platform cookies. Anyone running this tool with the optional MySQL storage pointed at a shared, hosted, or hijacked database is exposed.

  • highGHSA-r397-ff8c-wv2gaiomysql allows arbitrary access to client files through vulnerability of a malicious MySQL server
warninghigh · pypifixed in 48.0.1

cryptography 45.0.7

4 advisories. Fixed in 48.0.1.

How it can be abusedThis library handles encryption and certificate checking for the crawler's HTTPS connections and login-state handling. An attacker positioned on the network, such as someone running a hostile Wi-Fi network between you and a platform like Xiaohongshu or Douyin, could potentially exploit flaws in its bundled crypto to weaken that connection protection or slow the process down with malicious certificates, affecting anyone running the scraper.

  • highGHSA-537c-gmf6-5ccfVulnerable OpenSSL included in cryptography wheels
  • highGHSA-g6cj-pr64-35w5cryptography: PKCS#7 EnvelopedData decryption exposes a Bleichenbacher oracle through distinguishable errors and timing
  • highGHSA-jwv3-5hgf-82wwpython-cryptography: Duplicate self-signed intermediates can cause exponential path-building
  • highGHSA-r6ph-v2qm-q3c2cryptography Vulnerable to a Subgroup Attack Due to Missing Subgroup Validation for SECT Curves
warninghigh · npmfixed in 4.18.0

lodash-es 4.17.21

1 advisory. Fixed in 4.18.0.

How it can be abusedThe flaw lives in lodash's template function, which can run arbitrary JavaScript if an attacker controls the text passed into it. In this repo lodash-es is only a helper library for the React web dashboard, which builds its own pages and never feeds user or scraped data into templates, so an attacker has no realistic path to exploit it; users running the crawler are effectively unaffected.

warninghigh · pypifixed in 6.1.0

lxml 6.0.0

1 advisory. Fixed in 6.1.0.

How it can be abusedIf the crawler ever parses an XML feed or response using lxml's iterparse or ETCompatXMLParser defaults, a malicious server it scrapes could craft a document that makes the crawler read local files on the machine running it, such as config files with database passwords or saved login cookies. Anyone running MediaCrawler against untrusted or attacker-influenced sources could be exposed.

  • highGHSA-vfmq-68hx-4jfwlxml: Default configuration of iterparse() and ETCompatXMLParser() allows XXE to local files
warninghigh · pypifixed in 1.3.12

mako 1.3.10

2 advisories. Fixed in 1.3.12.

How it can be abusedThese bugs let someone read files outside the intended folder by passing crafted paths like backslashes or double slashes to Mako's template lookup, but only if the app renders templates from user-supplied names. MediaCrawler does not use Mako for templating anywhere in its crawling or web UI stack, so there is no realistic way for an attacker to reach this code.

  • highGHSA-2h4p-vjrc-8xpqMako vulnerable to path traversal via backslash URI on Windows in TemplateLookup
  • highGHSA-v92g-xgxw-vvmmMako: Path traversal via double-slash URI prefix in TemplateLookup
warninghigh · npmfixed in 3.3.16

nanoid 3.3.7

2 advisories. Fixed in 3.3.16.

How it can be abusedThis flaw only triggers if the app's own code calls nanoid with a zero or negative size, causing an infinite loop and freezing whatever runs it, such as the Node.js signing scripts or the web UI build tooling. An attacker cannot cause this remotely by themselves; only a developer mistake in how nanoid is invoked would hang the process, so users running crawls could see things stall but there is no direct attack path here.

warninghigh · npmfixed in 8.5.12

postcss 8.4.47

2 advisories. Fixed in 8.5.12.

How it can be abusedAn attacker would need to trick a developer or build process into compiling a malicious CSS file that contains a crafted sourceMappingURL comment; PostCSS would then read and expose arbitrary .map files from the machine running the build. In this repo PostCSS is only used to process the project's own trusted stylesheet sources during the optional WebUI build, so end users of the crawler are not exposed.

  • highGHSA-6g55-p6wh-862qPostCSS: Arbitrary file read and information disclosure via attacker-controlled sourceMappingURL in CSS comments
  • highGHSA-r28c-9q8g-f849PostCSS: Path Traversal in Previous Source Map Auto-Loading (sourceMappingURL) leads to Arbitrary .map File Disclosure
warninghigh · npmfixed in 3.29.5

rollup 4.22.0

2 advisories. Fixed in 3.29.5.

How it can be abusedAn attacker who can influence the WebUI build, such as through a malicious dependency or crafted file path, could get code written outside the build folder or inject script content into the bundled JavaScript served to anyone using the MediaCrawler web dashboard.

warninghigh · pypifixed in 1.3.1

starlette 0.37.2

3 advisories. Fixed in 1.3.1.

How it can be abusedAnyone who can reach the FastAPI web dashboard's HTTP endpoints could flood it with huge form or multipart upload bodies that Starlette fails to cap, hanging or crashing the crawl-monitoring server while crawls run. The Windows-only file-serving flaw applies only if you expose Starlette's static file handler on Windows, which this project generally does not.

  • highGHSA-82w8-qh3p-5jfqStarlette: request.form() limits silently ignored for application/x-www-form-urlencoded enable DoS
  • highGHSA-f96h-pmfr-66vwStarlette Denial of service (DoS) via multipart/form-data
  • highGHSA-wqp7-x3pw-xc5rStarlette: SSRF and NTLM credential theft via UNC paths in StaticFiles on Windows
warninghigh · pypifixed in 2.6.0

urllib3 2.5.0

4 advisories. Fixed in 2.6.0.

How it can be abusedThe crawler sends requests carrying your platform login cookies, so if any target server or proxy responds with a malicious redirect chain, this bug can leak those cookies to an unrelated site, letting an attacker hijack your Xiaohongshu or Douyin account; it can also send endless compressed data that hangs or crashes the crawl process.

  • highGHSA-2xpw-w6gg-jr37urllib3 streaming API improperly handles highly compressed data
  • highGHSA-38jv-5279-wg99Decompression-bomb safeguards bypassed when following HTTP redirects (streaming API)
  • highGHSA-gm62-xv2j-4w53urllib3 allows an unbounded number of links in the decompression chain
  • highGHSA-qccp-gfcp-xxvcurllib3: Sensitive headers forwarded across origins in proxied low-level redirects

Code review

warningauthzAll /api/* HTTP routes and /api/ws/* WebSocket routes served by api/main.py (crawler_router, data_router, websocket_router), bound to 0.0.0.0:8080

All /api/* HTTP routes and /api/ws/* WebSocket routes served by api/main.py (crawler_router, data_router, websocket_router), bound to 0.0.0.0:8080

An attacker gains full read access to all crawled data, can drive arbitrary crawler subprocesses on the victim's machine, and can harvest the victim's platform login cookies from logs.

Fix: Bind the server to 127.0.0.1 by default and add an authentication dependency (e.g. a FastAPI dependency requiring a bearer token from an environment variable) applied to every router via app.include_router(..., dependencies=[Depends(verify_token)]) and to both WebSocket endpoints.

Get this report every week for your repos.

GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.

$19 a month, flat · First 10 outputs free · No card required