Rebuilding a RT Datastore: the design, deployment & performance story
summary
Migration from the legacy key-value store Keebler to Aerospike replaces a hard-to-update UDP system with a TCP-backed cluster, validated by running new Aerospike queries in parallel against the old store and checking correctness; the feature store hits ~1 ms median lookup and the bidding stack needs ~40 ms p99. Major work focuses on client-side latency: tuning timeouts, moving to Java 11, using an ELB mesh-seed hack, raising memory and disk watermarks, and increasing connection limits to 150k–200k to avoid churn. Cost drops on i3en and a batch cluster sustaining 3–4M lookups/sec complete the win.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:25About Quantcast
- 2:23Realtime bidding
- 3:15Engineering problems
- 5:00Presentation
- 7:00Team
- 7:44Spoiler Alert
- 8:00Balloons
- 8:56Timeline
- 9:56Feature Store
- 17:45Space saving tricks
- 22:54AWS issues
- 26:46Disaster recovery
- 27:45Client performance
- 28:16Debugging
- 28:48Client problems
- 29:35Connection limits
- 30:55Client performance improvements
- 32:23Frequency capping
- 33:58Keebler
- 36:16Key takeaways
- 38:20Aerospike performance
- 40:18Lessons learned
- 41:51Future plans
- 42:07Questions answers
related talks
36mHot Swap Your Datastore: A Practical Approach and Lessons Learned
18mProduct Review and Roadmap
33mInnovation with Aerospike: How AppLovin Handles Billions of Requests per Day
49mJourney with Aerospike: Architecture Best Practices
33mUsing Aerospike as a Real-time System-of-Record for Adobe Experience Cloud Services on Azure
22mThe Art of the Possible
26mPowering Real-time Decisions With a Reduced Footprint
29mWhat breaks when Aerospike hits 6 million QPS