Data in Motion: Streaming Static Data Efficiently in Akka Persistence

on YouTube ↗42m

" summary

Explains concrete work on the Akka Persistence Query implementation for akka-persistence-cassandra, showing how queries for events by persistence ID use cheap partitioned range reads, why events by tag requires materialized views, and how the initial replay wrapped blocking Cassandra calls while the new approach reuses the stream for non-blocking replay with configurable consistency. Highlights a pragmatic tradeoff — tag queries delay reads (default 10 seconds), dual‑write hazards with materialized views, and an alternative pipeline that merges per‑node write‑ahead logs to preserve causal ordering and enable efficient, correct stream materialization.

AI-written summary · about

" chapters (31)

1. 0:00Intro
2. 0:29Databases
3. 0:44Batch processing
4. 1:02Data at scale
5. 2:11Streaming static data
6. 2:31Pulling data from source
7. 2:53Inserts
8. 3:27Updates
9. 3:56Pushing data from source
10. 4:17Infinite streams of finite data source
11. 4:30Log data structure
12. 5:32Pulling data from a log
13. 8:07Akka Persistence Query Cassandra
14. 9:49Actor publisher
15. 11:15Events by persistence id
16. 11:58All persistence ids SELECT DISTINCT persistence_id, partition T
17. 13:06Events by tag
18. 17:13Akka Persistence Cassandra Replay
19. 18:27Non blocking asynchronous replay
20. 19:01Benchmarks
21. 19:52Alternative architecture
22. 23:40Event time processing
23. 25:07Ordering
24. 25:50Distributed causal stream merging
25. 27:54Exactly once delivery
26. 30:50Optimisation
27. 33:10Table and stream duality
28. 35:23Infinite streams application
29. 37:44Distributed systems
30. 40:29Challenges
31. 41:24Conclusion

" related talks

• 48mUnlocking Real-time Insights: Enhancing Your Databases With Stream ProcessingYingjun Wu · Big Data Conference Europe 2023
• 55mApache Cassandra — Streams and MemoryDmitriy Konstantinov · Joker 2021
• 39mModern Event-driven Architecture: Adopting Data in MotionAlex Stuart · GOTO EDA Day 2022
• 47mPatterns for Persistence and Streaming in Cloud ArchitecturesJeff Carpenter · Data Con LA 2019
• 01h 12mHidden pearls for high-performance-persistence in JavaSven Ruppert · Joker 2020
• 42mContemporary Approaches to Data at ScaleBen Stopford · Code Mesh 2015
• 45mData Modeling in an Event Centric WorldChris Condron · DevDays Europe 2022
• 31mApache Cassandra Analytics: A Recipe to Move Petabytes of DataDinesh A. Joshi · Cassandra Summit 2023