Large Scale Real Time Ad Invalid Traffic Detection with Flink

on YouTube ↗19m

" summary

Criteo processes 300 billion bid requests and 4 billion displays daily and moves its invalid‑traffic rules from Kafka Streams to Flink, claiming faster rule rollout via the Flink SQL API, native windowing, watermarks and checkpointing. Multiple Flink jobs consume Kafka, output detections to Kafka and feed a Mesos consolidator. A hidden bug with watermarks kept only in memory causes false positives and revenue loss, so watermark tracking is moved to an external watermark service on Mesos and timestamps are assigned at source, which fixes the issue.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 1:51What is Invalid Traffic
3. 3:44Where is the money
4. 5:51Solution
5. 7:17Kafka API
6. 9:00Framework
7. 11:22Design
8. 13:04Kafka

" related talks

• 40mReal-time Customer Engagement in Gaming Using Kafka and FlinkVolker Janz · GOTO Amsterdam 2023
• 29mA brief history of time with Apache FlinkMohamed ABDESSEMED, T. Lamirault · Flink Forward Berlin 2016
• 40mReal-time Customer Engagement in Gaming Using Kafka and FlinkVolker Janz · Big Data Conference Europe 2023
• 29mWhen your stream doesn't stream very wellNiels Basjes · Flink Forward Global Virtual 2020
• 20mReal-time analytics with Flink and Druid #bbuzzJan Graßegger · Berlin Buzzwords 2016
• 42mStream Processing with Apache FlinkTugdual Grall · DevFest Toulouse 2017
• 47mReal-time Fraud Detection Challenges and SolutionsFawaz Ghali · NDC Oslo 2023
• 38mHow to handle 100k events per secondNiels Basjes · Berlin Buzzwords 2019