Intelligent Serverless and Scalable Real-Time Data Pipeline using Kinesis, Fargate and CFN
summary
Shows a working pipeline built with Kinesis Data Streams, Fargate tasks, and a CloudFormation stack where producers push into a one‑shard stream and Firehose writes to S3. CodeBuild builds images to ECR, an ALB fronts producer containers, and consumers read via the KCL; locust fires ~78,000 requests and ECS scales producers from 1 to 3 tasks by a 25% CPU target. Demonstration proves end‑to‑end ingestion, autoscaling, and CloudWatch logging all function as claimed.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:28Serverless
- 4:16Streaming Data
- 11:43DataEng Chart
- 13:05Containers
- 15:29ECS
- 16:39Filegate
- 18:56Cloud Formation
- 19:47Demo Architecture
- 21:08Demo Cloud Formation
- 29:33Demo Dashboard
- 30:03Logs
- 33:53Changing number of tasks
- 35:43Conclusion
related talks
32mBuilding and scaling a log analytics platform - a serverless approach
38mServerless Data Streaming on AWS
47mServerless Compute at the Heart of Your EDA
40mServerless Compute at the Heart of Your EDA
39mBuilding Serverless Application Pipelines
47mMainframe & Serverless Integration
41mTechnology mix: Around the serverless in 180 days
42mBuild and automate a modern serverless data lake on