pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
How pathway is put together
Pathway is a Python ETL/stream-processing framework for batch and streaming data, LLM pipelines, and RAG. The public API is Python (pip package 'pathway'), but all computation runs on a scalable Rust engine built as a CPython extension module ('pathway.engine') via PyO3/maturin, based on Differential Dataflow and Timely Dataflow (vendored under external/). The Rust side implements the dataflow graph, incremental operators, persistence, licensing, and a large set of source/sink connectors (Kafka, Postgres WAL, MySQL binlog, MongoDB, S3/DynamoDB/Kinesis, Azure, Elasticsearch, Pulsar, NATS, MQTT, RabbitMQ streams, ClickHouse, Delta Lake/Iceberg, etc.). The Python layer provides pw.io connectors, stdlib transformations, an LLM/RAG xpack with vector stores and REST servers, and a monitoring dashboard.
Languages
Frameworks
Datastores
Infrastructure
Major components
Rust engine core (src/engine)
Implements the incremental dataflow graph, expressions, reducers, joins/windowing, time handling, and execution over Differential/Timely Dataflow.
Python bindings & API (src/python_api.rs, python/pathway)
Exposes the engine to Python via PyO3, including table operations, UDFs, schemas, and pw.run() entry point.
Connectors (src/connectors)
Readers/writers for Kafka, Postgres, MySQL, MongoDB, MSSQL, S3/AWS, Azure, Elasticsearch, Pulsar, NATS, MQTT, RabbitMQ, ClickHouse, QuestDB, filesystems, and data lakes (Delta/Iceberg), plus parsers/formatters (Avro, BSON, Debezium, JSON lines, CSV/DSV) and schema-registry codecs.
Persistence subsystem (src/persistence)
Snapshots pipeline state and connector offsets so pipelines can resume after restart or crash, with configurable persistent-storage backends.
Licensing module (src/engine/license)
Validates enterprise licenses (ed25519-dalek present for signature verification) gating features like unlimited workers/exactly-once.
Python stdlib library (pathway.stdlib)
Higher-level transformations including indexing (USearch KNN, Tantivy BM25, hybrid indexes), temporal primitives, and ML utilities.
LLM xpack (pathway.xpacks.llm)
LLM wrappers, embedders, splitters, document/vector stores, question answering, and REST servers (e.g., QASummaryRestServer, VectorStoreServer) integrating LangChain/LlamaIndex.
Tests & integration suites (tests/, integration_tests/)
Unit and end-to-end tests covering webserver/LLM xpack behavior, RAG evaluation accuracy, index memory bounds, and connector correctness against real services.
A quiet week focused on keeping the daily examples fresh.
5 weeksWeek of 2026-08-24
Week of 2026-08-24latest
A quiet week focused on keeping the daily examples fresh.
Week of 2026-08-17
Pulsar output support landed alongside faster UDFs and pandas 3 compatibility.
Week of 2026-08-10
New Apache Pulsar connectors arrived, plus big wins for resilience and dependency freshness.
Week of 2026-08-03
Tooling week: Python 3.14 readiness, faster tracing, and CI repairs.
Week of 2026-07-27
Two releases shipped (0.32.0 and 0.32.1) along with major Rust tooling upgrades.
Over the past five weeks, the Pathway team shipped two releases (0.32.0 and 0.32.1), added brand-new Apache Pulsar input/output connectors with Avro support, and broadened compatibility with newer Python (3.14), pandas 3, and updated dependency versions. Alongside these features, substantial effort went into reliability: faster Python UDF execution, persistence improvements that favor duplication over data loss, and a steady stream of test hardening and CI fixes to keep builds green.
Week by week
2026-08-24A quiet week focused on keeping the daily examples fresh.latest1 change
Chore
Daily Pathway examples refresh
The repository's example code was automatically refreshed twice during the week.
2026-08-17Pulsar output support landed alongside faster UDFs and pandas 3 compatibility.5 changes
Feature
Avro output format for Pulsar
Data can now be written out to Pulsar in Avro format with automatic schema registration.
Feature
pandas 3 support
Pathway now works with both pandas 3 and the older pandas 2, so users aren't forced to upgrade or downgrade.
Refactor
Faster Python UDF hot path
Custom user-defined functions written in Python now run noticeably faster on each record.
Fix
Clearer Pulsar credential errors
When a Pulsar broker rejects a connection, error messages now point users toward their credentials.
Fix
QuestDB retries and test cleanup
QuestDB connections retry more gracefully, telemetry thread counts are limited, and tests were tidied up.
2026-08-10New Apache Pulsar connectors arrived, plus big wins for resilience and dependency freshness.6 changes
Feature
Apache Pulsar connectors
Pathway can now read from and write to Apache Pulsar message queues as inputs and outputs.
Fix
Safer persistence for non-replayable sources
When replaying saved data isn't possible, the system now prefers duplicating messages over losing them.
Fix
Kafka consumer stays polled under backpressure
The Kafka consumer keeps receiving messages even when downstream processing slows down, avoiding stalls.
Fix
TCP connections can no longer freeze pipelines
Socket waits are now time-bounded so a stalled network connection can't hang the whole data pipeline.
Chore
Dependency modernization
Python and other dependency version ranges were relaxed to admit the newest stable releases, including numpy 2.5.
Chore
Test stability improvements
Stream, Kafka, and S3 tests were made independent of timing quirks and broker hiccups that caused flaky failures.
2026-08-03Tooling week: Python 3.14 readiness, faster tracing, and CI repairs.6 changes
Feature
Python 3.14 support
Release and verification pipelines now build and check against the upcoming Python 3.14.
Refactor
Faster expression/operator trace capture
Internal bookkeeping of how expressions and operators are traced was sped up.
Fix
Dependency resolver guidance for protobuf/grpc
Pip is now steered past compatibility issues between new protobuf-6 and grpc releases.
Fix
mypy type checks fixed across Python versions
Type-checking jobs were repaired and now run correctly on Python 3.10 as well.
Docs
Improved pull request template
The template contributors fill out when proposing changes was made clearer.
Chore
CI unbreaking and timeouts raised
Broken workflow files were fixed and build/test job timeouts were increased to reduce spurious failures.
2026-07-27Two releases shipped (0.32.0 and 0.32.1) along with major Rust tooling upgrades.6 changes
Feature
Pathway 0.32.0 and 0.32.1 releases
Two new versions of Pathway were cut and published during the week.
Chore
Rust upgraded to 1.97 with refreshed crates
The underlying Rust compiler and its library dependencies were brought up to current versions.
Refactor
PyO3 upgrade to 0.29.0
The bridge between Python and the Rust engine was upgraded several minor versions ahead.
Docs
New contribution policy
Contribution guidelines were rebalanced for an era of cheap AI-assisted code generation, with the PR template adapted to match.
Fix
Flaky S3 credential tests fixed
Tests that were being poisoned by leaked environment variables from another library now run reliably.
Fix
macOS build workaround propagated
A workaround for a specific hardware issue was applied to release and universal2 macOS builds.
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
No concrete code-level issues confirmed.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required