Blazing fast I/O of data in the cloud with Daft Dataframes
summary
Daft (www.getdaft.io) gives a concrete end-to-end claim: a Rust I/O layer with a Python API that actually loads massive numbers of cloud files fast. Benchmarks show 44 GB parquet in ~30s with 9 Gbps peaks, projection pushdown to two columns in 3s, predicate pruning to 2–2.5s on S3, glob listing of 10,000 files in 300 ms versus boto3’s ~2.5s, 40k JPEGs (≈13 GB) in ~20s, and a TPCH 1 TB Q1 run peaking ~7.5 Gbps; deployed at Amazon reading petabytes and reporting ~38% cost efficiency.
AI-written summary · about
chapters (14)
- 0:00Daft introduction
- 1:31What is Daft
- 5:25Why Daft was built
- 6:05Cloud I/O performance
- 7:31Demo setup and metrics
- 10:23Reading Parquet files
- 11:53Column projection pushdown
- 13:19Row predicate pushdown
- 14:23Optimizing S3 file listing
- 16:27Reading small CSV files
- 18:33Processing images from URLs
- 19:57TPC-H terabyte query
- 21:02Answering audience questions
- 22:29Closing thoughts and future work
related talks
23mDaft - The Distributed Python Dataframe for Complex Data
38mScale out from the very beginning
30mSpeed is Not All You Need for Data Processing
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
29mEfficient and portable DataFrame storage with Apache Parquet
15m100x faster data app experiences delivered cost-effectively
40mScaling Python - Bank Edition
19mAccelerating ML Development with Multi-Modal Datasets: Python, Parquets, and Daft