The Future of Data Pipelines with Atomic Wasm Transformations & the Evolving Role of Data Engineers
summary
A new protocol implements atomic Wasm transformations that take a JSON-like input and return a JSON object compiled into a compact CBOR/SIBO array, chains steps by indexed ports, and uses content-addressed CIDs stored on IPFS. A working Holium CLI is shown compiling Rust to Wasm, importing a CSV, connecting two transformations, running the pipeline and producing a cleaned CSV with reformatted country names and codes. Source, specs and code are published on GitHub.
AI-written summary · about
chapters (4)
related talks
26mHarnessing WebAssembly for Real-time Stateless Streaming Pipelines
28mDistributed Computation with WASM and WASI
24mReaching the Unix Philosophy's Logical Extreme with Webassembly
05mClosing Remarks
21mBring Your Own Bytecode to the Logging Party
56mKEYNOTE: WebAssembly Update: The Proposals, Wasi, And The Component Model
43mMeet Your Future - WebAssembly
49mWasmer Things: An Upside-Down Guide to WebAssembly