Designing a Cloud-Native Data Pipeline with AWS
summary
Designing a cloud-native, serverless data pipeline on AWS that can scale while controlling costs. It covers choosing and arranging AWS services across six domains: data ingestion, storage, cataloging, processing, consumption, and security/governance, using only AWS-managed serverless services.
AI-written summary · about
chapters (16)
- 0:00Opening
- 1:03Why this talk?
- 3:58Talk objectives
- 6:00Why use AWS?
- 10:23Retrieving data from our competitors
- 19:51Retrieving competitors' data
- 25:08Summary
- 26:02Services
- 28:17Parallelism
- 29:27Optimization
- 31:32Governance
- 34:02Automation
- 36:16Using the same language
- 36:50Recap
- 38:29Conclusion
- 39:48Q&A
related talks
34mDesigning Data Intensive Applications In Serverless Architecture
49mServerless Architectural Patterns and Best Practices
40mBuilding scalable end-to-end deep learning pipelines in the cloud
49mBuilding an Enterprise Data Lake Architecture on AWS
41mBuilding end to end streaming analytics serverless pipeline
42mBuild and automate a modern serverless data lake on
25mA simple serverless data pipeline
43mManaging a Data Lake on AWS