Building Data Pipelines on AWS
summary
Practical claims and concrete recipes decide tonight: cost traps that produced an unexpected $500 bill, optimization rules for Athena like Parquet + partitioning reduces scanned TB and cost, architecture recipes such as kinesis → firehose → s3 → redshift/elasticsearch for streaming analytics, serverless savings like 1 million free Lambda invocations and EMR step jobs to avoid 24/7 clusters, plus an AWS Batch scraper that parses hundreds of thousands of pages in under an hour.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:57What is a data pipeline
- 3:20Processing steps
- 8:00ETL versus pipelines
- 9:44Pipeline architectures
- 14:19AWS data engineering services
- 25:40Architectures and AWS services
- 27:07AWS Glue vs Data Pipeline
- 28:11Batch pipeline example
- 30:17Streaming pipeline example
- 32:03AWS Batch for computation
- 33:47Summary
related talks
31mHow (Not) to Spend $15000 per Month on AWS Athena
46mWhen the CFO Comes Knocking: Our Journey to a Cost Efficient Data Platform
32mBuilding and scaling a log analytics platform - a serverless approach
01h 02mCentralised Logging without the blood, sweat and tears
57mCloud Adoption Patterns
27mMastering AWS Cost Optimisation
19mUltra-Budget AWS: Running Scalable Apps for Pennies
38mAccelerating innovation through AWS Cloud technology