Collecting Big Data at Scale with Amazon Kinesis

on YouTube ↗55m

" summary

Shows how to ingest clickstreams and CloudTrail logs into Kinesis Data Streams or Kinesis Data Firehose, using either the Kinesis Producer Library or a Kinesis agent, with microbatching and Lambda consumers. Documents concrete limits: one shard ≈1 MB/sec write (≈1,000 records/sec) and 2 MB/sec aggregate read, retention 24 hours–7 days, shard split/merge planning (doubling capacity can take hours). Demonstrates a working pipeline: CloudTrail → Firehose → S3 + Data Analytics → Kinesis Stream → Lambda → DynamoDB dashboard.

AI-written summary · about

" chapters (20)

1. 0:00Introduction to big data
2. 3:14Real-time vs near real-time
3. 7:05Kinesis vs Apache Kafka
4. 12:24Kinesis producer library
5. 15:34Kinesis agent for logs
6. 17:43Kinesis client library
7. 20:13Serverless architecture demo
8. 23:25Kinesis data stream setup
9. 27:25Kinesis data analytics
10. 30:15Lambda function processing
11. 32:40Event-driven pubsub model
12. 35:55Distributed system challenges
13. 39:03Enhanced fan-out benefits
14. 41:15Shard scaling strategies
15. 42:39Kinesis data firehose
16. 45:30Evolutionary architecture
17. 46:15Flight booking case study
18. 51:18Netflix use case
19. 53:09Firehose configuration
20. 54:34Design principles summary

" related talks

• 35mStreaming and Real-time AnalyticsDonnie Prakoso · AWS Summit Online EMEA 2020
• 39mLarge scale streaming analyticsJay Palaniappan · Data Con LA 2019
• 35mStreaming and Real-Time AnalyticsDonnie Prakoso · AWS Summit Online 2020
• 38mServerless Data Streaming on AWSAnahit Pogosova · Code Mesh LDN 2020
• 29mThe need for speed: Data streaming architecturesRajeev Chakrabarti · AWS AMER Summit May 2021
• 53mDefine data streaming architectures & solutions to fit your needsEd Berezitsky · AWS Summit Atlanta 2022
• 26mArchitecting real-time transactional data lakeMasudur Rahaman Sayem · AWS Summit Sydney 2025
• 23mCustomer Journey with streaming data on AWSRahul Pathak · Flink Forward Europe 2019