Low Latency Serving of Offline Data: Efficient, Safe, and Reliable Data...
summary
Preprocesses offline Iceberg data into RocksDB SST files, stage them in S3, then load those files onto dedicated serving nodes with lease-based partitioning and hash-mod routing so cutovers are near-instant and validated. Claims include 50 TB in 30 minutes (vs a week), 99% faster deploys and P99 <5 ms. Uses partition-aware sizing, NVMe vs EBS tradeoffs, CI/CD-style validation, blue/green flips and instantaneous DNS rollback to reduce cost and operational risk while enabling high replication and rapid iteration.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:01Serving offline data challenge
- 2:50Writing to mutable stores fails
- 5:19Freshness, cost, risk trilemma
- 6:01Reconstructing known data wasteful
- 7:45Data deployment like code
- 8:50Stateless API layer stateful
- 10:57Stateful systems operational challenges
- 12:00Leasing and partition-aware ownership
- 13:24Capacity planning with chunky instances
- 14:49Partition-aware sizing algorithm
- 16:59NVMe versus EBS cost trade-offs
- 18:23Routing requests to nodes
- 19:47Hiding complexity with abstractions
- 21:13CI/CD for data with blue-green
- 24:21Configurable rollback trade-offs
- 25:24Decoupling serving and loading
- 26:48Latency and throughput results
- 29:16Key lessons and benefits
- 31:45Q&A
related talks
18mOpen-Source OLAP on Open-Source Data Warehouses.
25mOffline-first data: Getting Bigger
33mNext Gen of Datastores: Hot Queries, Cold Storage
28mAchieving Rapid Response Times in Large Online Services
39mHow we isolate streaming ingest from search using RocksDB
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
50mDesigning a scalable data platform based on Apache Spark/Iceberg
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales