How to ingest 100 billion events from millions of devices?
summary
Explains how to ingest 100 billion events per month from millions of devices by building a scalable data pipeline. Covers technologies and architecture for high-throughput ingestion, device connectivity, batching/streaming approaches, and operational practices to handle volume and scale.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:52Data ingestion pipeline
- 7:20Request reception and caching
- 10:52Event transformation and storage
- 14:46Real-time data aggregation
- 18:15Data Lake storage and querying
- 26:46Security and infrastructure monitoring
- 31:17Cost management and optimization
- 33:19Terraform deployment
- 35:26Architecture pros and cons
- 38:39Q&A
related talks
50mHow We Store 60,000 Events per Second
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
55mCloud Analytics: from 0 to 100,000 events per second in 50'
36mHow to Build a Solution Processing Terabytes of Data (...)?
24mGive meaning to 100 billion analytics events a day, analytics at Teads
22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
44mProcess and analyze millions of events per second with open source tools
33mManaging IoT Storage with Google's Cloud Platform