Real-Time Streaming Segmentation on Azure
summary
Real-time streaming segmentation on Adobe RT CDP runs on Aerospike + Azure with a hub/spoke design, query decomposition, incremental state stored in Aerospike, and namespace-based multitenant isolation. The system achieves about 100x throughput per sandbox (from ~1.5k to ~150k QPS), global peaks ~500k events/sec, and latency drops from ~300 ms P95 to ~20 ms P99, reduces ops burden, stabilizes IO, and successfully serves large-scale real-time audiences.
AI-written summary · about
chapters (12)
- 0:00Real-time streaming segmentation intro
- 2:09Adobe Experience Platform overview
- 3:54RT CDP and core problem
- 6:24Segmentation workloads explained
- 10:31Workload trade-offs
- 12:57Real-time segmentation anatomy
- 14:40First system challenges
- 15:21Solution with Aerospike
- 18:48Architecture and multi-tenancy
- 20:13Cluster provisioning & throughput
- 21:15Operational results & improvements
- 23:00Future workload expansion
related talks
33mUsing Aerospike as a Real-time System-of-Record for Adobe Experience Cloud Services on Azure
44mApache Spark on Azure - solution (...)
08mReal-Time Data Systems: Speed, Scale, and Resilience
25mWhy it Takes a Team of Engineers to Process Events in RT
35mArchitectures and Trends for Event Stream Processing on Azure with Open Source Software
28mAccelerating Real-Time Analytics with Cosmos DB and GPU-Enhanced Serverless Apache Spark
30mFive Challenges in Real-time Stream Processing and Five Solutions
48mStreaming Azure Services