vaex
High-performance Python library for out-of-core DataFrames on big tabular data.
High-performance lazy Out-of-Core DataFrames for big tabular data exploration and visualization.
Vaex is a high-performance Python library designed for lazy out-of-core DataFrames to visualize and explore big tabular datasets. It targets data scientists and engineers who need to process billion-row datasets without wasting memory or immediately needing a cluster.
What it is
It calculates statistics on N-dimensional grids for over a billion samples per second using memory mapping, zero memory copy policy, and lazy computations. It supports fast groupbys, subsecond joins, and expression transformations without materializing intermediate data on disk.
Who it's for
It suits data professionals working with massive tabular files in HDF5 or Apache Arrow formats on local hardware. It does not suit teams looking for traditional in-memory Pandas workflows without adapting to lazy evaluation paradigms.
Key features
Instant File Opening
Open huge HDF5 and Apache Arrow files instantly via memory mapping.
Lazy Expression System
Transform data lazily on the fly without wasting memory or time on feature engineering.
Out-of-Core Processing
Keep data untouched on disk, streaming only when needed to delay cluster requirements.
Subsecond Joins
Join massive tables quickly without copying or materializing the right table.
Up and running in 3 steps
Our honest verdict
What's great
- Processes over a billion rows per second on local hardware
- Zero memory copy policy prevents wasted RAM
- Integrates into Jupyter and Voila for interactive dashboards
- Performs fast parallelized groupby operations on categorical data
Where it falls short
- Requires data to be in specific formats like HDF5 or Apache Arrow for best performance
- Remote DataFrame support is still developing
- Requires learning Vaex-specific idioms instead of standard Pandas patterns
Pricing & license
Free under its MIT licence, with commercial support and data science solutions available through vaex.io.
Open remote AI-training roles
873 open remote AI-training roles, updated November 15, 2026; most pay $20–$170 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
These reviewed AI tools do the same job instantly — from our scored directory.
Use it, or have us set it up
FAQ
What file formats does Vaex support?
Vaex natively supports HDF5 and Apache Arrow formats for memory-mapped, instant file opening.
Can Vaex replace Pandas entirely?
Vaex provides a similar API to Pandas for out-of-core data, but relies on lazy evaluation and specific file formats for performance.
