Introducing Arc: A common int. lang. for unified batch and stream analytics.
summary
Arc implements a common intermediate representation that cross-compiles pandas, Beam and other frontends into a single dataflow and enables optimizations across libraries. The team builds a native Arcon Rust runner and generates Flink-compatible code; current frontends include pandas and Beam frontends. They demonstrate operator fusion on long chains (a 50-map chain and larger experiments) showing a large reduction in per-operator invocation overhead, and support partial-evaluation and dataflow fusion to produce much faster unified batch and stream execution.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:13Data Movement Costs
- 2:52Common Intermediate Representation
- 5:06Compilation Pipeline
- 9:53Black Box Problem
- 10:46Task Fusion
- 13:40Conclusions
- 14:27Experiments with the Beam Integration
related talks
39mTaming the language border (...)
09mBackstage interview
03mRay: A Distributed Execution Framework for Emerging AI Applications
33mBridging Big Data and Machine Learning Ecosystems: A Cloud Native Approac...
29mUnified batch and stream processing in python
39mFlare & Lantern: Accelerators for Spark and Deep Learning
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
34mLeveraging Apache Arrow for ML Workflows