A new approach to DataFrames and pipelines
summary
Vaex claims a working DataFrame that handles billion‑row tables on a laptop by memory‑mapping columnar storage (Apache Arrow / hdf5) and keeping an immutable computational graph of expressions. It opens massive files instantly and computes virtual columns without copying using JIT or CUDA acceleration, runs aggregations and maps over a billion rows in seconds, and exports the pipeline state so trained models and transforms become reproducible automatic ML pipeline saved as JSON. It also supports S3 streaming and remote execution.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:08Python data science ecosystem
- 2:37What is Effects
- 4:26Handling large datasets
- 5:53Opening 1.2 TB file
- 6:38New York taxi dataset
- 9:11Live demo begins
- 13:44Mapping pickup and drop-off
- 14:48Feature engineering
- 17:18Machine learning with EffectsML
- 21:30Building pipelines
- 24:43Testing pipeline
- 25:03S3 streaming support
- 26:10Remote billion-star computation
- 26:53Conclusion and key takeaways
- 28:42Q&A
related talks
23mDaft - The Distributed Python Dataframe for Complex Data
39mComplex Data Transformations Made Easy
34mLeveraging Apache Arrow for ML Workflows
18mData Preprocessing for ML at Production Scale
39mAccelerating Data Science with RAPIDS
40mBeyond Pandas: The Great Python Dataframe Showdown
33mBridging Big Data and Machine Learning Ecosystems: A Cloud Native Approac...
29mStructuring Data for Scalable AI Applications