Five Challenges in Real-time Stream Processing and Five Solutions
summary
Hazelcast-based pipeline demo connects Redpanda (Kafka) topics, generates orders with Python, joins streams in an in‑memory IMAP and runs SQL-like queries to emit real‑time recommendations; demo shows enrichment working live and explains tiered storage and state checkpoints to disk for reliability. Claims: handle 1 billion events and meet strict latencies like 50 ms SLA for fraud checks by keeping data close to compute, using 3‑node coworker clusters and persisting last‑processed offsets to survive failures.
AI-written summary · about
chapters (10)
- 0:00Introduction to stream processing
- 3:29Why stream processing matters
- 7:45Real-time latency challenges
- 13:08Real-time vs historical data
- 16:20Hazelcast platform overview
- 18:26Kafka and microservices scaling
- 21:12Blue-green deployment model
- 22:38Live demo with Hazelcast
- 25:53Open source community resources
- 26:15Q&A session
related talks
49mEnriching Kafka Applications With Real-time Contextual Data
08mReal-Time Data Systems: Speed, Scale, and Resilience
19mSupercharging Kafka Applications with Real-time Contextual Insights
35mStreaming and Real-Time Analytics
33mReal-time Trade Data Processing and Machine Learning at Giga-Sca...
53mIntroduction to data streaming
35mStreaming and Real-time Analytics
50mThe Real-Time Analytics Stack