Build an in-place Amazon S3 data lake with
summary
MongoDB Atlas Data Lake lets you query S3 in place (no ETL or moving data), mapping S3 prefixes to MongoDB namespaces and running full MQL aggregation pipelines across JSON, CSV, Parquet, ORC and even Excel. It uses IAM role + STS temporary creds for secure read-only access, a distributed query router and regional agents that do parquet predicate pushdown and per-file parallelism. Demo builds a data lake in minutes, returns aggregation results live, and is in public beta since June.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:07Modern business data is complex
- 1:35The cost of storing data keeps dropping
- 3:44Hadoop challenges
- 4:54Cloud data lakes emerge to deliver the new data lake
- 7:37Characteristics of good data lakes
- 8:42Making data easy is what MongoDB is all about
- 9:59MongoDB Atlas - Global cloud database
- 13:41Atlas Data Lake architecture
- 16:31Data organization
- 17:19Data formatting
- 18:20Amazon S3 Server access logging
- 21:07Trust relationships
- 21:45IAM role assumption
- 24:55Data Lake configuration
- 26:25Data Lake federation
- 38:17Atlas Data Lake backend MOL
- 38:49Lifetime of a query: $group
- 41:00Atlas Data Lake security
- 45:16MongoDB Atlas Data Lake
related talks
15mSave The World And Money with MongoDB Data Lakes
19mBuilding modern data lakes on Amazon S3
16mBreakdown data silos - Build a data lake on Amazon S3
34mBest Practices for Implementing a Data Lake in Amazon S3
30mHarness the power of your data with the AWS Lake House Architecture
21mBreakdown Data Silos: Why Amazon S3 Is The Best Place To Build Your Data Lake
47mConstruyendo Lagos de Datos y Analíticos en AWS + Datos Abiertos
14mBuild your data lakehouse with Starburst Galaxy