Build a Modern Data Platform on AWS
summary
Builds a Lakehouse on AWS using Delta Lake on S3, streaming via MSK + Kafka Connect into Apache Spark on EMR, with compaction twice daily and vacuum every three hours; runs ~2,000 source tables (30–40% used), targets a 30‑minute data lag, and separates storage from compute via Redshift Spectrum plus QuickSight. Uses AWS Glue for catalog and schema registry, Lake Formation for row/column access control, two Airflow deployments (MWAA) and a Jarvis automation to onboard tables; the platform is working and horizontally scalable.
AI-written summary · about
chapters (13)
- 0:00Modern data platforms on AWS
- 4:45What is a data platform
- 5:49History of data architecture
- 7:58Lakehouse architecture benefits
- 10:57Core AWS data platform components
- 12:03Data Lake design zones
- 15:14Data acquisition challenges
- 19:38Redshift and QuickSight for data
- 22:54Airflow and EMR pipelines
- 27:35Glue and Lake Formation metadata
- 30:06Metadata governance strategies
- 32:40Future improvements and observability
- 35:54Conclusion
related talks
47mBuilding Data Lake Platform Fully Serverless
25mBuild open format data lakehouse
47mConstruyendo Lagos de Datos y Analíticos en AWS + Datos Abiertos
29mData Acquisition: Overcoming Challenges
53mHow to build a modern data architecture
31mFrom Warehouse to Lakehouse: Data at Rest
42mModern Data Strategy on AWS
39mBuild Your Data Lake on AWS — The Flo Experience