~/talk

Data Caching for Enterprise-Grade Petabyte-Scale OLAP

on YouTube 20m
Talkat USENIX ATC 2024 · 2024-07-10..07-12· [databases]

speakers

name:Chunxu Tang
links:none on file
name:Bin Fan
links:none on file
name:Jing Zhao
links:none on file
name:Chen Liang
website:http://crazydonkey200.github.io/
name:Yi Wang
links:none on file
name:Beinan Wang
links:none on file
name:Ziyue Qiu
links:none on file
name:Lu Qiu
linkedin:luqiu-ai
name:Bowen Ding
links:none on file
name:Shouzhuo Sun
links:none on file
name:Saiguang Che
links:none on file
name:Jiaming Mai
links:none on file
name:Shouwei Chen
links:none on file
name:Yu Zhu
links:none on file
name:Jianjian Xie
links:none on file
name:Yutian (James) Sun
links:none on file
name:Yao Li
links:none on file
name:Yangjun Zhang
links:none on file
name:Ke Wang
speakerdeck:kevinwang15
name:Mingmin Chen
links:none on file

resources

video:https://youtu.be/TXMkhyX8V8Q
slides:none on file

summary

Alluxio local cache embeds on-node SSDs and a page store to serve reads for Presto and HDFS, cutting network I/O and API calls. Deployed at Uber and Meta for three years, it reduces scan-phase wall time by about 67% P90 and 64% P50, serves over 70% of reads in one-hour windows, and limits traffic to external storage to under 10% (HDFS hot-block tuning drops misses toward 1%). Operational lessons: 1 MB pages, metadata caching, rate limiting, and fallbacks for corruption and full-SSD cases.

AI-written summary · about

chapters (9)

  1. 0:00Petabyte-scale OLAP caching
  2. 1:27Why start caching project
  3. 4:21Cache system requirements
  4. 6:31Alexy local cache architecture
  5. 8:40Cache design choices
  6. 11:11Uber Presto use cases
  7. 14:49HDFS node caching at Uber
  8. 16:37Lessons learned trade-offs
  9. 18:05Unexpected failures solutions

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.