Large Scale User Behavior Analytics by Flink

on YouTube ↗43m

" summary

A real-world UBA pipeline runs Flink as the streaming engine, handles 10k+ unique users across 10+ dimensions, and targets bursts near 100k TPS. CEP logic initially runs in Drools (memory pressure from facts causes failures), so RocksDB-backed Flink state, savepoints and rule-jar hot deploys on Flink 1.4 are used; patterns get grouped or split across jars to avoid thousands of initialized streams, and updates cause only second-scale downtime.

AI-written summary · about

" chapters (24)

1. 0:00Introduction
2. 0:34Haos background
3. 1:18Agenda
4. 2:20User Behavior Analytics
5. 4:36Challenges
6. 6:13Why Flink
7. 7:57Architecture
8. 10:16Rules Engine
9. 11:10Jewels
10. 12:43Jewels Pros Cons
11. 14:16ETL Needed
12. 15:34Hard Deployment
13. 16:21Jules Rule
14. 18:15Short Term
15. 19:37Problems
16. 20:46External Key Value Store
17. 21:12Time to Live
18. 21:51Optimizing Jewels
19. 23:03Plan B
20. 25:03Workflow
21. 27:39Optimization
22. 29:09Pros Cons
23. 31:08Demo
24. 36:54Questions

" related talks

• 45mApache Flink — Real-Time Data ProcessingGaël Renoux · Devoxx France 2024
• 38mStreaming analytics and CEP ... #bbuzzTill Rohrmann, Fabian Hueske · Berlin Buzzwords 2016
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 20mBuilding metric platform using Flink for massive scale at NetflixAbhay Amin · Flink Forward Virtual 2020
• 38mApache Flink: A Very Quick GuideVladimir Bacvanski · Data By the Bay 2016
• 18mReal-time Experiment Analytics at Pinterest with Apache FlinkParag Kesar, Ben Liu · Flink Forward Europe 2019
• 34mA journey of scaling high-volume low-latency stateful processingArti Pande · Flink Forward Global Virtual 2020
• 37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?Stephan Ewen · Flink Forward Global Virtual 2020