Best Practices for Implementing a Data Lake in Amazon S3
summary
Argues that Amazon S3 is the storage foundation for scalable data lakes and shows concrete customer results: Viper processes 10–15 billion events/day (peaks ~300k events/sec) via Kinesis/Firehose, EMR for hourly parquet conversion, and sees Athena querying performance improve; FINRA ingests ~75 billion market events/day with ~50k files/day and uses EMR, Presto/Redshift to route queries; details S3 features (storage classes, lifecycle, batch ops, S3 access points, 3,500 puts / 5,500 gets per-prefix/sec) and demos Lake Formation converts to Parquet and enforces column-level access with Glue and Athena.
AI-written summary · about
chapters (10)
- 0:00Introduction to Data Lakes
- 4:56Data Lake Foundations
- 6:48Best Practices and Design
- 9:31Customer Architecture Case
- 13:23Architecture Design Patterns
- 17:01Amazon S3 Optimization
- 20:38Security Best Practices
- 23:31Introduction to Lake Formation
- 25:16Lake Formation Demonstration
- 32:34Conclusion and Resources
related talks
34mBest Practices for Implementing a Data Lake in Amazon S3
19mBuilding modern data lakes on Amazon S3
43mManaging a Data Lake on AWS
21mOptimize and manage data on Amazon S3
16mBreakdown data silos - Build a data lake on Amazon S3
31mBest practices for building your data lake on Amazon S3
21mBreakdown Data Silos: Why Amazon S3 Is The Best Place To Build Your Data Lake
30mHarness the power of your data with the AWS Lake House Architecture