Build Your Data Lake on AWS — The Flo Experience
summary
Flo migrated from a monolithic data warehouse to an S3-based data lake, ingesting data with Kinesis/MSK and using AWS Glue for cataloging while initially querying with Athena/Presto. At scale (800M–3B rows/month and single tables ~400GB/month) Athena failed, so Flo switched to a self-hosted 25-node Presto cluster on EMR that reads S3 + Glue metadata and uses Parquet columnar files, which reduced query failures and costs (EMR ~ $4k/month) while keeping raw data in S3/Glacier.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:13Data architecture evolution
- 7:39Data lake concept
- 9:03Flo data processing
- 12:09Message queues role
- 15:44Amazon S3 storage
- 16:55AWS Glue catalog
- 19:10Amazon EMR processing
- 20:36Amazon Athena analytics
- 23:06Flo data challenges
- 26:29Data lake migration issues
- 28:37Presto for Flo analytics
- 30:48Platform building tips
- 32:13Adapting and future plans
related talks
47mConstruyendo Lagos de Datos y Analíticos en AWS + Datos Abiertos
37mBuild a Modern Data Platform on AWS
47mBuilding Data Lake Platform Fully Serverless
25mBuild open format data lakehouse
40mThe Journey to Implementing a Data Lake
13mBuild your open data lakehouse with Dremio and AWS (PRT085)
31mFrom Warehouse to Lakehouse: Data at Rest
44mBig Data with AWS: our lessons learned using managed services