Process and analyze millions of events per second with open source tools
summary
A practical walkthrough of building a high-throughput real-time analytics pipeline using open-source tools: Apache Kafka for event streaming, QuestDB for low-latency queries, Grafana for live dashboards, and Jupyter for interactive analysis and prediction. The talk demonstrates ingestion code in Python, JavaScript and Go, explains why Kafka remains necessary alongside a fast database, and includes demos and a GitHub project you can use as a starting point.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 5:19Real-time data problems
- 11:44Architecture evolution
- 14:55Open-source streaming stack
- 17:25Kafka data ingestion
- 23:46Efficient data formats
- 25:30Kafka Connect integration
- 28:13Fast databases
- 31:42Massive ingestion demo
- 34:09Grafana visualization
- 38:23Real-time machine learning
- 40:50Component monitoring
- 42:56Summary and resources
related talks
47mBuilding a Real-Time Analytics Database
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
06mJay Kreps interview
01h 29mEvent stream processing using Kafka Streams
24mData analytics with distributed tracing data
47mBuilding Modern Data Streaming Apps with Open Source
30mArchitecting Applications With Multiple Open-Source Big Data Technologies