Shaping the Future of Real-time Data Pipeline
summary
Claims a simple path to run batch code as streaming from day one and demonstrates a working serverless pipeline: Python transformation code deployed to glass flow receives Supabase webhooks, calls OpenAI for price suggestions, and outputs to a webhook UI (ten events processed in the demo). Contrasts Java-heavy stack (Debezium and Kafka, Flink) and their nine‑month rollout stories with Python frameworks, cites 170 languages for the translation app, and shows monitoring, retries, and dependency handling.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:37Real-time pipelines matter
- 5:07Real-time vs streaming data
- 6:30Bob’s price recommendation
- 9:21Kafka and Flink challenges
- 12:14Infrastructure and language barriers
- 17:52Python frameworks for real-time
- 21:10Serverless event-driven pipelines
- 24:02Request vs event-driven architectures
- 27:17AI with event-driven pipelines
- 30:10Demo: real-time car price pipeline
- 42:19Summary
- 43:02Q&A
related talks
39mStreaming doesn’t have to be hard
10mDemocratizing the pipeline with Apache Beam and Dataflow
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
42mWeb Development To Big Data. A Journey
41mAdding Generative AI to Real Time Streaming Pipelines
28mReal-Time Data for Generative Feedback Loop (GFL) Automation
47mAdvances in Stream Analytics: Apache Beam and Google Cloud Dataflow deep-dive
41mPython, Java, or Go: It's Your Choice ...