Unified batch and stream processing in python
summary
Claims a single Python library lets one codebase run both batch and microbatch streaming jobs: use beavers to orchestrate a DAG of functions, use Apache Arrow as the in‑memory tabular format to avoid slow pandas concat, and read from/write to Kafka replay or files for live and backfill modes. Demonstrations show zero-copy concatenation and fast Arrow→pandas views, state vs ephemeral nodes, and microbatch strategies for latency and correctness.
AI-written summary · about
chapters (12)
- 0:00Unified batch and stream intro
- 1:26Batch feature calculation example
- 5:21Batch vs stream differences
- 7:08Stream job challenges in Python
- 9:13Stream to batch conversion
- 13:10Apache Arrow for data
- 16:23Orchestrating streaming jobs
- 18:51Introducing Beavers
- 21:41Building Beavers DAG
- 23:08Beavers with Kafka
- 24:34Replay mode for testing
- 25:17Conclusion
related talks
36mKafka Streaming, the Pythonic Way
39mStreaming doesn’t have to be hard
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]
44mShaping the Future of Real-time Data Pipeline
38mData processing with Apache Beam
20mStreaming DataFrames: A New Way to Process Streaming Data
27mReal-time Event Streaming with Python