Build and automate a modern serverless data lake on
summary
Shows a fully automated serverless data-lake pipeline that reduces ETL burden and operational cost by using S3, Glue, Lambda, CloudWatch Events as the four core components. Demonstrates crawling and cataloging (crawler finds ~99% partition similarity), auto-generating Glue ETL code, and converting sample CSV taxi data into Parquet, deployable from a CloudFormation template that creates two buckets, two Lambdas, SQS/SNS and crawlers. Pipeline sends completion email and makes data queryable in Athena/QuickSight, claiming faster time to value.
AI-written summary · about
chapters (6)
- 0:00Introduction to modern data lakes
- 9:41Data ingestion methods
- 25:59Automating with AWS Lambda
- 30:15End-to-end serverless pipeline demo
- 39:37Building the serverless data lake
- 41:52Q&A and session wrap-up
related talks
49mBuilding an Enterprise Data Lake Architecture on AWS
47mBuilding Data Lake Platform Fully Serverless
34mDesigning Data Intensive Applications In Serverless Architecture- 47mBuilding Data Lake Platform Fully Serverless
25mA simple serverless data pipeline
43mManaging a Data Lake on AWS
25mData Gone in 60 Seconds: A Serverless ETL Heist
47mConstruyendo Lagos de Datos y Analíticos en AWS + Datos Abiertos