Case Study of Moving from CRUD to Log Based State Management

on YouTube ↗51m

" summary

Claims a move from CRUD to an append-only log stored as MongoDB documents, using two-layer logs—a lightweight metadata log pointing to a huge business event log—to store aggregates, tombstones and 2D timestamps. Handles up to one billion records, deduplicates by business-time vs processing-time, shards queries across partitions, and cuts heavy initial loads from about 30 hours to 4.5 hours by parallelizing dataset runs and replaying or deactivating batches for recovery. Retention is per-flow.

AI-written summary · about

" chapters (12)

1. 0:00Intro
2. 2:58Context: Master Data Management (in 1 min)
3. 4:40Context: Manufacturing Master Data
4. 6:13Motivations for Rethinking Data Management
5. 11:32Fragility of incremental CRUD Batch Processing
6. 14:55Logging Application Events Event Sourcing
7. 20:36Rethinking the Batch Dataset
8. 29:27Composition of Partial State as an Event Stream
9. 32:28MongoDB as an Event Log
10. 37:39Deriving State from the Event Log
11. 40:20Functional Schematic
12. 46:31Final Thoughts

" related talks

• 32mStoring State Forever: Why It Can Be Good For Your AnalyticsYaroslav Tkachenko · Flink Forward Global 2021
• 29mWide Event AnalyticsIgor Wiedler · LISA19
• 38mPostmortem: Evolution of an Event recording solutionMichael Robellard · PyOhio 2016
• 11mBig Data Analytics: Immutability & IdempotenceKnut Nesheim · GOTO Berlin 2014
• 14mSolving Data Engineering ChallengesSteve Lee, Joel Roland · DataEngBytes 2020
• 45mReal Time Analytics with MongoDBIngo Dyck, André Peglow · Confitura 2013
• 48mIntegrating microservices by accepting the data dichotomyDamien Gasparina · Voxxed Days Microservices 2018
• 30mData @Scale Seattle- Sundaram NarayananNarayanan Sundaram · Data @Scale Seattle 2015