How We Store 60,000 Events per Second
summary
An engineering talk about storing and processing a stream of 60,000 events per second. It describes the system architecture, storage technologies and techniques used to ingest, persist and query high-throughput event data (e.g., databases, queues, sharding/partitioning and batching) to achieve that write rate and maintain performance.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 11:15Old solution problems
- 14:08New storage choice
- 15:13Current architecture
- 20:54Key implementation approaches
- 29:55Results and performance
- 49:26Q&A
related talks
43mHow to ingest 100 billion events from millions of devices?
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
22mIt’s all about the db - a journey to accommodate 100k events per second
38mHow to handle 100k events per second
39mHigh-volume Data Streaming and Storage
28mBest of Stream + Batch: When Streaming Analytics Isn’t
48mHyper Speed! When Big Data Blooms
53mWhat it takes for stream data processing