Best Practices for Implementing a Data Lake in Amazon S3
summary
Practical checklist and hard numbers: Amazon S3 is the storage foundation, customers store and query petabytes using Glue, EMR, Athena and Redshift while converting to Parquet/ORC for speed; use 16–256 MB objects, multiple prefixes, and expect ~3,500 PUT/5,500 GET per-second per-prefix. Viber: 10–15 billion events/day, peaks ~300k events/sec; uses Kinesis Firehose → S3 then EMR to compact to Parquet; Athena outperforms Presto. FINRA: 75 billion market events/day, 50k files/day, ~0.5T validations/day; hybrid querying routes to Presto or Redshift by complexity. Demo: Lake Formation registers S3, Glue crawler builds catalogue, column-level grants enforced and Athena queries succeed.
AI-written summary · about
chapters (11)
- 0:00Introduction and agenda
- 1:11Data growth and scale challenges
- 6:37Why choose Amazon S3 for data lakes
- 9:35Data lake foundation best practices
- 15:19Lift and shift vs managed services
- 17:11Amazon S3 storage classes and lifecycle
- 19:23Performance design patterns for S3
- 20:48Security best practices for data lakes
- 23:19Introducing AWS Lake Formation
- 25:34AWS Lake Formation demo walkthrough
- 32:33Key takeaways and conclusion
related talks
34mBest Practices for Implementing a Data Lake in Amazon S3
16mBreakdown data silos - Build a data lake on Amazon S3
21mOptimize and manage data on Amazon S3
19mBuilding modern data lakes on Amazon S3
21mBreakdown Data Silos: Why Amazon S3 Is The Best Place To Build Your Data Lake
59mModern data lakes with Lake Formation, Amazon Redshift & AWS Glue
43mManaging a Data Lake on AWS
28mTurn Data Into Insights