A Quest to Synchronize Analytical and Operational Stores in Real Time
summary
Describes a practical walkthrough of synchronizing analytical (data warehouse) and operational (OLTP/real-time) stores for high‑load systems. Covers concrete techniques and architectures for keeping analytics up to date in real time in large data environments, including handling high write volumes and consistency between stores.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:11Domain overview
- 4:43Sync problem statement
- 7:15Sync approach evolution
- 8:40Stage 1: Direct queries
- 11:30Stage 2: Batch exports
- 14:03Stage 3: CDC adoption
- 15:27MongoDB CDC details
- 17:59MongoDB Connector usage
- 22:52API vs connector comparison
- 26:05CDC event format
- 28:33CDC solution architecture
- 31:01CDC processor features
- 33:10Monitoring and resilience
- 34:15Implementation results
- 39:16Q&A
- 53:12Conclusion
related talks
05mThe Future of Data Warehousing
58mUnderstanding Big Data for Software Engineers
34mReal time OLAP and Stream Processing Friends or Foes
49mAnalytics in SQL Server and Azure SQL Database
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
09mLet's Get Real: Acting on Data in Real Time
52mData Warehouse, Data Lake, Data Mesh, Oh My: The History of Data Storage
47mExpert Panel: Current & Future State of the Art in Real-Time Data Infrastructure