Building Data Lake Platform Fully Serverless
summary
Built a self‑service, fully serverless data lake on AWS that ingests vendor and partner streams, keeps raw copies and near‑real‑time events, and exposes data via APIs and Athena queries. Core pieces are Kinesis Firehose, Lambda, S3, DynamoDB, EventBridge, Glue, Athena plus API Gateway signed‑URL uploads and ServiceNow/GitHub onboarding automation. It supports 60 internal products and hundreds of integration pipelines, stores hundreds of GB of text and ~1 TB including binaries within 6 months, reduces costs with S3 lifecycle and crawler scheduling, and works but bumps into Glue/Athena scan costs and Lake Formation limits with open‑table formats.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:34Problem statement
- 10:38Choosing data lake
- 16:00Core architecture
- 20:21Landing zone design
- 28:51Processing and storage
- 33:23Self-service platform
- 36:48Key takeaways and solution architecture insights
- 39:59Cost optimization
- 42:07Scaling challenges
- 44:35Alternatives comparison
- 46:20Data lake vs big data
related talks
47mBuilding Data Lake Platform Fully Serverless
42mBuild and automate a modern serverless data lake on
37mBuild a Modern Data Platform on AWS
34mDesigning Data Intensive Applications In Serverless Architecture
32mBuilding and scaling a log analytics platform - a serverless approach
01h 00mProcessing a Data Lake on Node.js in a Serverless Architecture
47mConstruyendo Lagos de Datos y Analíticos en AWS + Datos Abiertos
49mServerless Architectural Patterns and Best Practices