Stream SQL with Flink @ Yelp

on YouTube ↗22m

" summary

Yelp runs a service called Stream SQL built on the Flink SQL API that wires SQL queries into the existing Kafka + Avro + schema-registry pipeline, converting Avro records to Flink Rows via a custom StreamTableSource that exposes processing/event time and emits retraction (update) streams. Tests run in Docker with a help-compose stack. About forty production queries run in the past year, the largest at ~140,000 msgs/sec. Current limits: Flink 1.6 placement bug, no changelog/table-stream source, and coarse window-trigger control.

AI-written summary · about

" chapters (10)

1. 0:00Introduction to Yelp streaming
2. 1:02Streaming at Yelp
3. 2:20Flink infrastructure evolution
4. 5:22Data pipeline architecture
5. 6:25Adopting stream SQL
6. 9:15Implementation details
7. 12:21Output and retraction logic
8. 14:09Developer experience and tests
9. 17:18Challenges and roadmap
10. 20:43Q&A session

" related talks

• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Berlin Buzzwords 2021
• 23mFrom Streams to Tables and Back Again: A Demo of Flink’s Table & SQL APITimo Walther · Flink Forward Berlin 2017
• 43mUnified stream and batch processing with Apache Flink’s relational APIsFabian Hueske · Flink Forward Berlin 2017
• 21mStream Processing and SQLBruno Bonnin · ScalaIO 2018
• 32mThe easy way to build and run Flink SQL jobs using Cloudera SQL Stream BuilderKenny Gorman · Flink Forward Global 2021
• 31mWriting a Streaming SQL & Materialized View EngineKenny Gorman · Flink Forward Virtual 2020
• 42mStream Processing with Apache FlinkTugdual Grall · DevFest Toulouse 2017
• 40mThe Seven Deadly Sins of StreamingGiannis Polyzos · Big Data Conference Europe 2025