Building a Real-time Ingestion Platform using Kafka and Aerospike
summary
DBS Bank builds a pipeline that ingests events to Kafka, projects partial updates into Aerospike as a compact Avro blob keyed by customer id, and persists hourly/day snapshots to S3 in Parquet/Carbon for cost savings. A YAML-based globalizer DSL maps source models to a unified enterprise model, a real-time snapshot holds current-day records, Presto queries federate Aerospike plus file backs, and end-to-end latency target is ~250 ms while maintaining eventual consistency.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:35What is our platform
- 5:17Batch ingestion
- 5:44Source system evolution
- 6:21Challenges
- 9:01Source systems
- 10:41Shadow database
- 12:54Choosing a solution
- 18:23T2 Layer
- 20:08Globalizer
- 21:14Storage
- 22:29Components
- 23:42Aerospike
- 28:10Data Platform Challenges
- 30:14Conclusion
related talks
48mThe real-time database to build your AI future
08mBuilding a Real-time Data Future
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
42mBuilding a near real time personalized feed using data streaming and Aerospike
29mData Acquisition: Overcoming Challenges
34mLarge Scale NoSQL DB Migration Under Fire
33mInnovation with Aerospike: How AppLovin Handles Billions of Requests per Day
32mStreaming Analytics with Apache Pulsar