Networks! project - real-time analytics that controls 50% of mobile network in Poland
summary
Real-time ingestion and analytics platform processes 2.2 billion messages a day from 750 Kafka topics to compute more than 5000 KPIs and 1500 SQL-defined aggregations for half of Poland’s mobile cells, built on Flink/Flink SQL with Ververica and Kubernetes, HDFS for raw storage and ClickHouse for monitoring. Deduplication uses Kafka key hashing and exactly-once Flink semantics; RocksDB tuning, NVMe disks and bloom filters raise throughput to 8–10k events/sec per thread. NetStream operator automates job lifecycle, versioned upgrades and reprocessing; JDBC sink backport handles Oracle ingestion.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:09Facebook outage
- 3:52Network operators
- 5:25Data warehouse
- 8:01Oracle switching
- 9:43Stream processing questions
- 10:15What in networks
- 11:01Aggregation
- 12:56Current state
- 13:31Data flow
- 14:13Duplication
- 15:20Challenges
- 17:28Aggregating
- 19:31Bugs
- 22:26Data piping
- 23:19Change management
- 23:41Job deployment
- 24:09Upgrade process
- 25:01Future plans
related talks
42mStream Processing with Apache Flink
42mHow Netflix Built a Real-Time Distributed Graph for Internet Scale
34mChasing the Stream Processing Utopia
32mBuilding Real-time Systems with Open Source Technologies
20mReal-time analytics with Flink and Druid #bbuzz
40mThe Seven Deadly Sins of Streaming
58mRealtime Analytics Geofencing Kafka
43mChange data capture in production with Apache Flink