Colliding worlds or bending Spark towards enterprise needs
summary
Claims that a Spark-based pipeline can serve enterprise CQRS-style needs by routing commands through Kafka into Spark Streaming processors that call business handlers and persist to MongoDB. Prototype uses a Spring facade, Docker dev environment, 100 ms micro-batches, and shows cold insert ~460 ms and warm ~25 ms; adds an event callback for synchronous acknowledgement. Notes checkpoint/serialization caveats, a lower-bound latency (~20 ms) for this approach, and that it works but trades strict low-latency guarantees for throughput and scalability.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:08Relativity theory and Spark
- 2:13CQRS in enterprise
- 3:59Secures and architecture
- 5:21CQRS model split
- 9:17App demo
- 12:03Docker infrastructure
- 13:47App launch and demo
- 25:50Data consistency problem
- 27:14Async solution
- 33:12Code changes for sync
- 35:22Summary and results
- 36:24Spark and Kafka pitfalls
- 38:30Command identification problem
- 39:35Spark guarantees and reliability
- 40:16Conclusion and Q&A
related talks
34mGoing deep with Spark Streaming #bbuzz
25mCONTINOUS APPLICATIONS WITH SPARK 2.0
42mDistributed Real-Time Stream Processing: Why and How
26mOptimal Approaches for Real-Time Machine Learning with Apache Spark on Kubernetes.
49mRethinking Reactive Architectures
01h 21mStructured Streaming with PySpark and Azure Databricks
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
47mReal-time Fraud Detection Challenges and Solutions