Building a Self-Service Streaming Platform at Pinterest

on YouTube ↗36m

" summary

Pinterest standardizes a Flink-based self-service streaming platform running Flink on YARN that replaces ad-hoc Kafka Streams and Spark micro-batches, handling terabytes per day and over a trillion Kafka messages companywide. Built features include a history-server tweak that embeds YARN application IDs and taskmanager→container mappings so web links go straight to YARN log pages, and a backfill system that replays S3-backed Kafka backups (Merced files) and uses a savepoint trick to transfer operator and source offsets from backfill to live Kafka; the approach works but restart is manual today.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 1:07Background on Pinterest
3. 3:59Choosing a streaming engine
4. 6:31Use cases for Flink
5. 12:19Challenge one log access
6. 22:46Challenge two backfilling
7. 34:41Future plans and summary

" related talks

• 29mA brief history of time with Apache FlinkMohamed ABDESSEMED, T. Lamirault · Flink Forward Berlin 2016
• 46mScaling PinterestMarty Weiner · GOTO Copenhagen 2014
• 35mKeynote: Stream analytics made real with Pravega and Apache FlinkSrikanth Satya · Flink Forward Virtual 2020
• 34mKeynote | Flink Ahead 2.0: The SequelKonstantin Knauf · Flink Forward Global Virtual 2020
• 25mMixing data streams at scaleBen Gamble · Aerospike Summit 2022: Mission Now
• 43mBuilding a community fountain around your data streamMaria Patterson · PyData Seattle 2017
• 42mHow Netflix Built a Real-Time Distributed Graph for Internet ScaleAdrian Taruc · Big Data Conference Europe 2025
• 37mMeasuring 2.0 – Building the next generation webanalytics ...Niels Basjes · Berlin Buzzwords 2016