A Community Cache with Complete Information
summary
Kariz is a shared community cache that sits next to the datalake, extracts DAGs and runtime signals from Pig/Hadoop and Spark, and centrally schedules prefetches and evictions. Prototype extends an existing distributed cache, implements three single-query algorithms (one prior, two novel) and three cross-query policies, and runs on a 16-node testbed. With the CMR algorithm it reduces TPC-H/TPC-DS job runtimes by up to 3× and cuts backend load by prioritizing maximally shared DAGs.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:43Deployment of analytic applications
- 2:05Caching for analytic frameworks
- 2:45Key insight: Community cache
- 3:59Community cache with complete information
- 6:21CMR: cache for minimizing runtime
- 7:56CMR: going beyond critical path
- 9:03Community cache implementation
- 9:37CMR evaluation
- 11:55CMR-M evaluation: compare with alternatives
- 12:49Community cache vs. cluster cache
- 13:49Conclusion
related talks
30mProject Caerus: Compute Storage Coordination
35mDagobah, a Data centric Meta scheduler
36mMoving Faster: Why Intent Media Chose Cascalog
35mHCatalog, Table Management for Hadoop
11mWhiz: Data-Driven Analytics Execution
15mOptimizing Query Performance in Big Data Systems
34mApache Iceberg.
40mApache Kudu (incubating): Fast Analytics on Fast Data #bbuzz