Petabyte Scale Datalake Table Management with Ray, Arrow, Parquet, and S3
summary
Amazon builds a production flash compactor using Ray, Arrow, Parquet, and S3 to apply upserts/deletes at petabyte scale, using partitioned primary key indexes and hash buckets to deduplicate and materialize data. Measured peak 24 TB/min on 8,000 vCPU (117 TiB input) and compacted a 1 PB partition on 27 TiB memory (≈12× larger than Spark), with average cost 46¢/TB (best 24¢/TB) and a 99.3% success rate, yielding a 91% cost reduction versus Spark on EMR.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:46Background
- 8:33Compact API
- 13:12Compacting S3
- 14:45Hash Bucket Step
- 15:46Parallel Hash Bucket Step
- 19:05materialized task indices
- 19:30materialized step
- 20:35recap
- 21:25results
- 24:42challenges
- 25:32summary
related talks
34mBest Practices for Implementing a Data Lake in Amazon S3
26mColumnar Storage: Redefining Data Management for the Modern Era
21mOptimize and manage data on Amazon S3
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake
22mTurn Data Into Insights
16mBreakdown data silos - Build a data lake on Amazon S3
22mAccelerating Analytics with Apache Arrow
20mGive me the Data