Define data streaming architectures & solutions to fit your needs
summary
Overview of AWS streaming choices with concrete tradeoffs: Amazon Kinesis Data Streams + Data Firehose for simple managed ingestion and delivery (now Kinesis Data Firehose supports direct sinks and up to a year retention) vs. Amazon MSK (managed Apache Kafka) and MSK serverless for lower-latency, offset-friendly workloads. Kinesis shows about 200 milliseconds end-to-end typical latency; some Kafka customers push toward ~20 ms. Schema Registry, MSK Connect (MirrorMaker patterns), replay via offsets, and scaling/partition cooldowns are demonstrated.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:30What is data streaming
- 5:05Use cases
- 9:38Data streaming
- 13:50Amazon Kinesis
- 14:47Kinesis Data Streams
- 16:14Apache Link
- 18:03Apache Kafka Managed Treatment
- 19:20Apache Kafka Connect
- 20:18Schemas
- 24:58Kafka vskinesis
- 27:28Region replication
- 32:48Latency
- 33:53Network
- 35:28Offsets
- 36:48Glue vs Apache Flink
- 40:06Errors in data streams
- 42:23kinesis vs Kafka
- 46:55Scaling operations
- 48:26cooldown period
- 49:18limitations
related talks
29mThe need for speed: Data streaming architectures
35mStreaming and Real-time Analytics
35mStreaming and Real-Time Analytics
23mCustomer Journey with streaming data on AWS
55mCollecting Big Data at Scale with Amazon Kinesis
39mLarge scale streaming analytics
26mArchitecting real-time transactional data lake
44mData Streaming: Which horse to bet on?