Yellowbrick: An Elastic Data Warehouse on Kubernetes
summary
An elastic Yellowbrick warehouse runs as containerized services on Kubernetes, persisting long‑term data in S3 and using local NVMe as cache with 100MB shards segmented to 2MB. The planner serializes plans to XML, compiles them to C++ and distributes object code to MPP workers; bulk loads hit ~10 TB/hour and streaming ingestion reaches 3–5M records/sec. Networking uses a reliable UDP via DPDK layer to cut TCP overhead, custom user‑space NVMe/network drivers and a fast allocator (claimed 10–100×) to avoid kernel copies. Scaling a provisioned cluster adds nodes in ~minutes.
AI-written summary · about
chapters (19)
- 0:00Start
- 1:29The Yellow Brick Software Stack
- 1:58Who and What Is Yellow Brick
- 3:45Hardware
- 4:28Use Cases
- 9:45The Cloud Data Warehouse Manager
- 14:59Workers
- 18:52Avoiding Kernel Overhead
- 20:07File Caching
- 21:11Are Cache Files Resilient Using Replication or Ratio Codes
- 22:08How Big Are Your Shards on S3
- 30:01Storage Engine
- 40:16Workload Management
- 41:05Resource Pools
- 43:04Workload Manager
- 43:23Flex Slots
- 44:20How Do You Predict How Many Resources Occurring Would Use before Running It and How Accurate Do these Predictions
- 46:45Os Level Optimizations
- 1:04:19How Does that Impact the Mapping of the Shards to the Instances of Your Compute Cluster
related talks
37mThe Evolution of a Planetary-scale Distributed Database
20mBuilding An Elastic Query Engine on Disaggregated Storage
19mBuilding KVStore for ML Workloads
33mBuilding an Elastic, Resilient Data Layer to Power New Retail Experiences
50mRunning Databases Efficiently at Scale - Case Study of Flipka...
16mPowering Smart Homes & Near Real-Time Cloud Agnostic, Distributed SQL
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse