Sharing what we love: The Apache Flink story at Pinterest

on YouTube ↗25m

" summary

Pinterest replaces Kafka Streams with Apache Flink to run large-scale stream apps that power features for 478 million users, claiming reduced latency from hours to seconds by adding a Flink speed layer that buffers 24 hours (and seven days for some flows) and supports backfill/reprocess inside the app. Two engineers operate a stats pipeline launched to wide usage; shopping catalog updates go from 24 hours to 5 seconds, real-time conversion prediction improves attribution coverage, and matrix aggregation and A/B testing move from flaky to production-grade.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to Pinterest
2. 3:57Pin stats use case
3. 8:27Content creator use cases
4. 14:39Shopping catalog use case
5. 20:02Infrastructure engineering use cases
6. 22:44Lessons learned with Flink
7. 24:09Closing remarks and Q&A

" related talks

• 18mReal-time Experiment Analytics at Pinterest with Apache FlinkParag Kesar, Ben Liu · Flink Forward Europe 2019
• 25mData: Moving with the flowAlex Walden · Big Data LDN 2022
• 21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & CassandraCiesielczyk, Zontek · Flink Forward Europe 2019
• 15mFeature Stores: Powering Machine Learning Infrastructure with Apache FlinkSimba Khadder · Flink Forward Global Virtual 2020
• 27mArchitecting Sub-Second Analytics PipelinesSai Kaushik Ponnekanti · Conf42 Machine Learning 2025
• 20mUsing Flink for Versatile Feature Engineering PipelinesMichal Ciesielczyk · Flink Forward Global Virtual 2020
• 18mThe role Stream Processing plays for an Opti Channel ExperienceJamie Chittenden · Flink Forward Europe 2019
• 15mData science at scale using Apache Flink: Dynamic model serving and real-time feature generationShashank Agarwal · Flink Forward Global Virtual 2020