Hot Swap Your Datastore: A Practical Approach and Lessons Learned
summary
Quantcast migrates a critical cookie datastore from keebler to aerospike with zero production downtime while keeping 99th‑percentile lookup latency near the existing ~80 ms target and handling ~1.5–1.8M rps. The process uses hybrid mode with 1% dual-read sampling and rack‑by‑rack canaries, sized clusters with nonblocking dark reads, and orchestration via Puppet and feature flags for fast rollback. Outcomes: reduced hosts from ~500 to 90, eliminated EBS IOPS tail spikes, but prototype blocking benchmarks mask async production issues that extend deadlines.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:30About Quantcast
- 7:14Why replace Keebler
- 12:00Aerospike
- 14:02Requirements
- 15:47Deployment Process
- 20:37Hybrid Mode
- 22:03Verification
- 23:20Running in Parallel
- 24:35Latency Performance
- 27:02Threepronged approach
- 29:07What went well
- 31:07Unpleasant surprises
- 33:08Audience takeaways
related talks
48mRebuilding a RT Datastore: the design, deployment & performance story
40mDeveloping highly scalable applications with Aerospike
34mLarge Scale NoSQL DB Migration Under Fire
26mPowering Real-time Decisions With a Reduced Footprint
23mMyntra's Sub-Millisecond Journey with Aerospike
06mSnapdeal and Aerospike
31mUser Attribution at Scale: All-Flash, ½ Trillion Objects
22mThe Art of the Possible