Challenges of Building Blazing Fast Data APIs
summary
Claims a working system called Dozer that turns CDC feeds into low‑latency, customer‑facing Data APIs using a single binary written in Rust and a realtime transformation engine that pre‑materializes queries, supports SQL TypeScript WebAssembly and ONNX model scoring, persists state with RocksDB and uses LMDB for fast caching, and auto‑publishes gRPC/REST endpoints with versioning and JWT security. It targets single binary multicore scaling on one machine and cites 99% of query workload is below 10 gigabytes.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:29Data APIs importance
- 6:09GraphQL caching challenges
- 9:22Real-time pipeline complexity
- 13:17Dozer simplifies APIs
- 16:53Rust and scaling
- 20:30Dozer open-source
- 23:48Real-time sourcing and processing
- 26:18Dozer streaming engine
- 30:34SQL and TypeScript transformations
- 35:32APIs analytics event apps
- 38:48Conclusion Q&A
related talks
16mEmpowering Fearless Data Engineering
05mBuilding Efficient Web Scrapers
13mLightning Talk: Revolutionizing Application Architecture with Wasm Powered Database Ex...
40mDuckDB: Harnessing in-process analytics for data science and beyond
46mAdopting Rust by gracefully oxidising web applications
25mWTF, Rust for Web Development??
28mRust and Scala, Sitting in a Tree….
47mRust At The Edge