The story of building a real-time streaming analytics system
summary
A retrospective on building a real-time stream analytics system, focusing on handling unbounded (infinite) data streams rather than finite datasets. Covers the problem of moving from batch-style, finite-data programming to continuous-stream processing and the design/implementation choices required to process ongoing request streams correctly.
AI-written summary · about
chapters (10)
- 0:00Business goals
- 2:58System demo
- 6:10Storage challenges
- 10:51Threading issues
- 15:33Frame pipeline
- 19:57Buffer management
- 24:57Reactive streams
- 31:03Frame rate & sampling
- 35:12Lazy vs greedy
- 37:41Q&A
related talks
47mOne streaming to rule them all: Streaming as the foundation of an analytics ecosystem
20mStreaming Architectures
53mWhat it takes for stream data processing
42mPrinciples and Patterns for Streaming Data Analysis
35mStreaming and Real-Time Analytics
32mBig Data Processing : Where do we go next ?
35mStandardisation and leveraging the exponential rise in streaming
50mStreaming Data: Don’t Miss the Train, Don’t Wait in Vain