When product meets real world data - how to analyze and tune deployments & upgrades
summary
Field report of a beta upgrade process that turns huge real-world data into predictable upgrades: clones of production author repos, migration S3 buckets, and checkpoints enable pre-extraction store from Lucene index, cutting store generation from days to hours; oak-run out-of-band indexing builds 6.4 indexes offline and merges them at catch-up, avoiding hours of live reindexing. Example customer has 100 TB repo / 4M assets with ~20 TB text-extractable and produced ~150–200 GB of index artifacts, shrinking downtime to a Friday–Saturday window.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 6:50Automation
- 9:20Datadriven decision making
- 10:55Active collaboration
- 12:10Conde Nast International
- 17:55Outofband indexing
- 24:00What is still open
- 24:15Whats next
- 27:13Takeaways
- 28:13Baseline
- 29:30Contribute
related talks
30mReliable Data Processing with Minimal Toil
06mNever ever upgrade
23mShift-Left: A DevOps and Database Story
34mZero Downtime Database Migrations and Deployments
42mScaling Data at Ovoko: The Succesful Strategy
51mPatterns & Playbooks to Make Every Commit Production Ready
41mScaling Observability: From Chaos to Clarity with Open Source Tools
36mHot Swap Your Datastore: A Practical Approach and Lessons Learned