Scalable Data Transportation & Ingestion
summary
MemQ is a Pub/Sub that offloads replication to object storage and stores data as packetized batches so ML training can ingest petabytes cheaply. It uses S3 as replicated storage and an async acknowledgment protocol to hide object-store latency while aggregating for compression. Production runs at GB/s, tolerates 2x spikes without expensive rebalancing, and reports 80–90% cost reduction versus Apache Kafka for ML ingestion; a low-latency variant reaches p100 <10s and the project is open source on GitHub.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:52ML data transportation challenges
- 3:15MemQ architecture
- 4:41MemQ acknowledgment model
- 6:24Why use batch-based storage
- 7:49MemQ vs Kafka scalability
- 8:31Kafka feature comparison
- 9:14MemQ optimizations
- 9:54MemQ impact and results
- 10:16MemQ future plans
- 11:18Key takeaways
related talks
05mDriving the On-Demand Economy with Predictive Analytics
05mClose Encounters with the Third Kind of Database
24mInference on Streaming Data at Scale at Intuit
57mOur DataMesh Journey: data architectures for microservices
37mLet's build our own message streaming platform
38mEnabling Big Data and Machine Learning for the Masses...
43mBuilding Scalable and Extendable Data Pipeline
25mPersistent and performant ingestion using Data Prepper