Kubernetes + Operator + PaaSTA = Flink @ Yelp

on YouTube ↗30m

" summary

Yelp runs roughly ~50 clusters, ~1,000 jobs and migrates Flink off AWS EMR to Dockerized Flink clusters on Kubernetes, managed by a Kubernetes operator that replaces slow EMR bootstraps (15 minute cluster startup) and yields far faster deploys while improving resource sharing. Operator integrates with PaaSTA integration so developers use familiar git push / pasta status / pasta logs workflows; a supervisor remains for compatibility but plans move operator-first (Beam, autoscaling, reactive mode).

AI-written summary · about

" chapters (11)

1. 0:00Intro
2. 0:33What is Yelp
3. 1:08Agenda
4. 2:51Connectors
5. 3:32Scale
6. 4:41Status quo
7. 6:07Problems
8. 7:22Meet Kubernetes
9. 9:11Flink Overview
10. 14:12Operator Overview
11. 22:49Flink Cluster

" related talks

• 42mManaging Apache Flink on Kubernetes - FlinkK8sOperatorAnand Swaminathan · KubeCon + CloudNativeCon North America 2019
• 37mEvolving Yelp's Streaming at ScaleAshish Khatkar · Berlin Buzzwords 2024
• 41mMore Bang for Your Buck: How Yelp Autoscales Mesos + Marathon on AWS Spot FleetRob Johnson · MesosCon Europe 2017
• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021
• 44mBuild a Real-time Stream Processing Pipeline with Apache Flink on AWSSteffen Hausmann · Flink Forward Berlin 2017
• 41mRunning Apache Flink everywhereStephan Ewen · Flink Forward Berlin 2016
• 32mStream Processing at Lyft with Flink and BeamSherin Thomas, Micah Wylde · Scale By The Bay 2018
• 30mDemystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about?David Moravek · Flink Forward Global 2021