Big Data Analytics in Near-Real-Time with Apache Kafka Streams
summary
Shows a working pipeline that pushes SQL Server transactions into Kafka via Kafka Connect, computes near‑real‑time user rewards with KSQL/Kafka Streams, and writes aggregates back to SQL Server and Elasticsearch. Demo runs in Docker Compose, uses Avro + schema registry, watches a rowversion column, seeds ~27,000 transactions across nine users, and sets topics to 5 partitions so stream apps can scale. Rebalancing moves partitions across instances and processing continues without data loss — the demo succeeds end‑to‑end.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 2:10Batch Processing Analogy
- 3:23Streaming Processing Analogy
- 6:23Traditional Database ETL
- 8:11Challenges in Modern Data
- 11:00What is Apache Kafka?
- 13:49Kafka Core Components
- 17:11Demo Architecture Overview
- 20:11Setting Up the Data Source
- 26:43Configuring Kafka Topics
- 30:50Understanding KSQL Objects
- 34:23Kafka Connect and Sinks
- 37:43Data Verification
- 39:01Real-time Aggregation
- 43:22Visualizing Data in Kibana
- 46:55Kafka Streams Application
- 52:11Parallel Processing Demo
- 56:32Q&A and Conclusion
related talks
01h 47mRadanalytics.io streaming applications workshop
42mHow to Leverage the Apache Kafka Ecosystem to...
40mHow to leverage the Apache Kafka Ecosystem to productionize Machine Learning
47mReal-time Fraud Detection Challenges and Solutions
19mAI Workflows with Kafka and Modern Architectures
42mWeb Development To Big Data. A Journey
34mActionable Insights with Real-time Streaming Analytics of Customer Reviews
19mBig Data