Streaming doesn’t have to be hard
summary
Shows a working stream-batch unified Python dataframe API that keeps transformation logic separate from execution, parses Python dataframe calls and compiles to an intermediate representation that can target Pandas, Flink, Spark, RisingWave or a GPU-native engine (Thesias). Demo runs a 10s tumbling window fraud-amount aggregation, compiles the same code to Flink SQL and inserts results into a Kafka topic, and also returns a Pandas dataframe via a DocDB backend. Team reports supporting 20+ backends and using SQLGlot to translate dialects while handling watermarks per-backend rather than in the IR.
AI-written summary · about
chapters (9)
- 0:00Introduction and welcome
- 1:47Batch vs streaming basics
- 5:37Machine learning pipelines challenges
- 11:32Stream-batch unification efforts
- 17:38Streaming API design approach
- 24:03Live demo of Ibis streaming
- 30:10Composable data systems vision
- 34:46How to use Ibis today
- 36:12Q&A session
related talks
39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
44mShaping the Future of Real-time Data Pipeline
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
21mStreaming DataFrames: A New Way to Process Streaming Data
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
43mA Data Streaming Architecture with Apache Flink #bbuzz
46mFlink SQL in 2020: Time to show off!
35mFlink's Table & DataStream API: A Perfect Symbiosis