Lessons Learned: Porting a Streaming Data Pipeline from Scala to Rust.
summary
Conviva ports a mission‑critical streaming pipeline (handling ~5 trillion events/day) from Scala/Akka to Rust and reports concrete wins: a prototype using careful data layouts and small‑vector techniques yields more than an order‑of‑magnitude speedup and about 12x less memory; the gateway port shows halves CPU vs Go and an order‑of‑magnitude memory drop. The new codebase implements a programmable DAG executor (dynamic operators, arenas, enum_dispatch, Tokio, custom actor layer) and proves both performant and production‑ready.
AI-written summary · about
chapters (12)
- 0:00Introduction and background
- 1:49Why port to Rust
- 5:22Porting goals and architecture
- 6:50Rust journey timeline
- 9:00Performance results overview
- 10:26Bringing developers on board
- 14:05Performance and data structures
- 16:36Stream processing design
- 20:30Static vs dynamic dispatch
- 23:59Async patterns in Rust
- 26:15Tooling and ecosystem
- 27:41Lessons and conclusion
related talks
16mEmpowering Fearless Data Engineering
34mStreaming video analysis in Rust using Pravega
28mRust and Scala, Sitting in a Tree….
55mScaling out operational research with Rust
40mStreamling: Lightweight, Extensible Streaming on DataFusion
46mAdopting Rust by gracefully oxidising web applications
19mSuki: Choreographed Distributed Dataflow in Rust
44mBending the Curve: How Rust Helped Us Write Better Ruby