Expert Panel: Current & Future State of the Art in Real-Time Data Infrastructure

on YouTube ↗47m

" summary

Panel contrasts two adoption paths: pumping fresher analytics into warehouses versus powering low-latency, stream-driven microservices for feature extraction and real-time scoring. Concrete technologies named include Apache Flink, Kafka, Google Dataflow and CDC bridges, and patterns like dynamic tables and sliding windows. Operational pain points are change data capture, checkpointing, state management and auto-scaling. Several teams report success with SQL-over-streams for expressiveness but still struggle with resource tuning, ordering guarantees and mixed batch/stream execution at large scale.

AI-written summary · about

" chapters (5)

1. 0:00Panel introduction
2. 2:51Current real-time use cases
3. 18:25Adoption barriers and challenges
4. 37:06Future of stream processing
5. 46:12Closing remarks and Q&A

" related talks

• 38mRealtime Data Activation - Analytics, Intelligence & Decision MakingMarcelo Wiermann · THE CONF 2017
• 08mReal-Time Data Systems: Speed, Scale, and ResilienceLakshmi Bhargavi Mullapudi · Conf42 JavaScript 2025
• 05mThe Future of Data WarehousingAnupam Singh, Brian Coyne · Strata Data Conference 2018 - New York, NY
• 10mData Rivers — The New Analytics ArchitectureRachel Pedreschi · OSA Con 2021
• 29mData Mesh Streaming Patterns: A Real-time PerspectiveAlok Pareek · Big Data LDN 2022
• 01h 14mPANEL - Streaming.Mary Grygleski, Dean Wampler, Zander Matheson, Oli Makhasoeva, Tim Berglund, Jerry Peng · Scale By the Bay 2023
• 59mPanel: Future of Data ServicesAlexy Khrabrov, William Morgan, Roman Shaposhnik, Neha Narkhede, Haoyuan Li, Marius Eriksen, Joel S. Horwitz · Data By the Bay 2016
• 42mTales from the Field: The Future is NowEllen Friedman · Jfokus 2019