CEP platform handling millions of users - lessons from 3 years in production
summary
A Flink-based CEP platform ingests Kafka streams from NiFi and on-prem Hadoop and now serves over 10 million subscribers, ~500k interactions/sec and billions of events/day, running hundreds of rules for marketing, fraud and automation with ROI under a year. Architecture starts with a shared-state single Flink job running hundreds of triggers then splits critical paths into separate jobs; key wins are a field-pruning operator before shuffle, a replayable incubating mirror job for production-only testing, and observability via Prometheus + Grafana + Loki; pain points are checkpoint timeouts, ordering across pipeline jobs and NiFi retries.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 0:59Kcell use cases
- 1:28System architecture
- 6:14Technical design
- 9:12Implementation details
- 16:49Project success
- 30:07Operations and monitoring
related talks
37mBuilding and operating large-scale streaming applications at King
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
42mApache Flink deep-dive #bbuzz
30mImplement Reliable, Isolated & Unified Job Submission
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
36mApache Flink in the Cloud-native Era
41mApplying Geospatial Analytics at a Massive Scale using Kafka, Spark and Elasticsearch on DC/OS
49mNetflix Keystone - Real-time Stream Processing Cloud Service