Building a near real time personalized feed using data streaming and Aerospike
summary
Company builds per-user precomputed feeds keyed by user id (one key per user) using Aerospike plus Kafka for ingestion and Arango for graph lookups, handling 35 million users / 700 million connections. Content gets ML enrichment asynchronously, stored with memory and disk namespaces, and updates use eventual consistency. A fanout shortcut treats authors with 1 lakh follower VIP threshold differently to avoid blasting all keys. Reads return sub-millisecond precomputed feeds; measured impact: reactions increase ~10%, affinity and impression metrics improve, and fanout update latency drops.
AI-written summary · about
chapters (13)
- 0:00Introduction to Koo and Aerospike
- 2:03Engineering for scale and micro worlds
- 4:48Content discovery and feed architecture
- 7:15Feed ranking, fan out, and notifications
- 10:19Architectural goals for content feeds
- 13:52User network density and matching
- 17:21Implementing Aerospike for performance
- 22:27Real-time feed composition
- 23:46Learning from architecture migration
- 26:02Impact and retention metrics
- 28:24System design best practices
- 30:26Future steps and knowledge graphs
- 31:44Q&A session
related talks
32mBuilding a Real-time Ingestion Platform using Kafka and Aerospike
08mBuilding a Real-time Data Future
32mStreaming Analytics with Apache Pulsar
40mMastering small chunks of data in high frequency ad-tech systems
33mInnovation with Aerospike: How AppLovin Handles Billions of Requests per Day
23mAdobe + Aerospike: Real-time Streaming Segmentation
32mFeed generation at ShareChat
33mExponential Growth Without Sacrificing Platform Speed