Streamlio: Towards a Unified End-to-end Solution
summary
A unified real‑time stack combines Apache Pulsar (multi‑tenant, low‑latency pub/sub) with Apache BookKeeper (durable, segment‑centric log storage) and Heron processing so one copy of streaming data serves live delivery and long‑term retention. The claim: segments distributed across bookies remove per‑partition disk limits, enabling arbitrary backfills and reprocess by offset without heavyweight ETL. Independent serving vs storage scaling, end‑to‑end encryption, geo‑replication and native multi‑language task isolation are implemented and shown to work at production scale.
AI-written summary · about
chapters (11)
- 0:00Introduction to real-time data
- 1:21Lambda architecture limitations
- 3:49Problems with patchwork systems
- 6:13Unified messaging and storage vision
- 10:31Pulsar and BookKeeper overview
- 17:46Pulsar’s unified messaging model
- 23:46Pulsar’s security and features
- 24:08Heron compute framework
- 29:25Real-time topology example
- 32:13Unified reprocessing benefits
- 33:18Q&A and closing
related talks
28mBuild your next-generation stream platform based on Apache Pulsar
45mScalable distributed messaging&streaming with Apache Pulsar
53mEvent Messaging and Streaming with Apache Pulsar
47mEvent Streaming for the Best of All Worlds
32mBig Data Processing : Where do we go next ?
48mApache Pulsar: Event Streaming in The Cloud Native World
47mChoosing the Right Messaging Platform for Your Event-Driven Application
30mEvent Streaming and the Future of Applications