pathway
Python ETL framework for stream processing, real-time analytics, and RAG.
Python ETL framework for real-time analytics, LLM pipelines, and RAG.
Pathway is a Python ETL framework powered by a scalable Rust engine for stream processing and real-time analytics. It suits professionals building live LLM applications and RAG pipelines using Python.
What it is
It executes Python data pipelines over batch and streaming data using a Differential Dataflow Rust engine. It includes connectors for Kafka, PostgreSQL, and GDrive, alongside specialized LLM helpers for real-time Vector Indexes.
Who it's for
It suits developers building live data feeds and RAG systems who want to code in Python while retaining Rust performance. It does not suit teams looking for zero-code UI pipeline builders.
Key features
Scalable Rust Engine
Executes Python code via a Differential Dataflow Rust engine for multithreading and distributed computations.
Unified Batch and Stream
Processes batch jobs, stream replays, and data streams using the exact same codebase.
LLM and RAG Tooling
Provides wrappers, parsers, embedders, and an in-memory real-time Vector Index for live document RAG.
Extensive Connectors
Connects to Kafka, GDrive, PostgreSQL, SharePoint, and over 300 data sources via Airbyte.
Up and running in 3 steps
Our honest verdict
What's great
- Same code base handles both local development, batch jobs, and real-time streams
- Built-in LLM extension includes embedders, splitters, and real-time vector indexes
- Handles out-of-order and late data points with automatic consistency updates
- Supports Airbyte integration for connecting to over 300 external data sources
Where it falls short
- Only available on MacOS and Linux natively, requiring a Virtual Machine for other operating systems
- Free version provides 'at least once' consistency while 'exactly once' requires an enterprise version
- Requires technical skill to set up Python environments, dependencies, and container deployment
Pricing & license
The repository does not state a licence name in its metadata, and the README mentions free and enterprise versions with differing consistency guarantees.
Open remote AI-training roles
879 open remote AI-training roles, updated November 15, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
Use it, or have us set it up
FAQ
What operating systems are supported?
Pathway is officially available on MacOS and Linux. Users of other systems must run it inside a Virtual Machine.
Does it support both batch and streaming?
Yes, the framework uses a unified engine allowing the same code to handle batch jobs, stream replays, and real-time streams.
How does it connect to external sources?
It includes built-in connectors for Kafka, GDrive, PostgreSQL, and SharePoint, plus an Airbyte connector for over 300 additional sources.
