Extending a High-performance Data Streaming System with WebAssembly
summary
Claims 60–70% of stream-processing needs move into the broker by running Wasm inside Redpanda (Seastar thread-per-core architecture). Implementation uses per-core VMs with preallocated linear memory pools, instruction-count yielding and cheap stack swaps (about nine CPU cycles) to respect a 500 μs task budget, avoiding reactor stalls. Developer flow uses rpk to build/deploy modules; current model copies each record into the VM (benchmarks show up to ~1+ GB/s in some tests) and provides at-least-once delivery with dead-letter options.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:42Red Panda engine
- 2:05Streaming transformations
- 3:52WebAssembly for transforms
- 5:15WebAssembly integration
- 7:00Developing WASM transforms
- 8:23XML to JSON in Go
- 10:31High-performance architecture
- 12:37Preventing reactor stalls
- 14:42CPU constraints for WebAssembly runtime
- 15:47WASM memory management
- 17:32WASM memory limits
- 19:59Key takeaways
- 20:46Q&A
related talks
26mHarnessing WebAssembly for Real-time Stateless Streaming Pipelines
21mAsync, bidirectional streams in WebAssembly
31mBuilding WASM Powered Distributed Stream Platform
39mFrom Event-Driven to Automotive
30mIntegrating Apache Arrow on WASM using Component Model
25mFrom Cloud to Edge computing - Unleashing the power of WebAssembly at the edge
32mWebAssembly on the Factory Floor: Efficient and Secure Processing of High Velocity Machine Data
21mHit the Ground Running with WebAssembly