~/talk

End-to-end I/O Monitoring on a Leading Supercomputer

on YouTube 25m
Talkat NSDI 2019 · 2019-02-26..02-28· [data-engineering, observability]

speakers

name:Bin Yang
links:none on file
name:Xu Ji
links:none on file
name:Xiaosong Ma
links:none on file
name:Xiyang Wang
links:none on file
name:Tianyu Zhang
links:none on file
name:Xiupeng Zhu
links:none on file
name:Nosayba El-Sayed
links:none on file
name:Haidong Lan
links:none on file
name:Yibo Yang
links:none on file
name:Jidong Zhai
links:none on file
name:Weiguo Liu
links:none on file
name:Wei Xue
links:none on file

resources

video:https://youtu.be/PGC7iuPag-0
slides:none on file

summary

Beacon implements end-to-end I/O tracing on the 40,960‑node Sunway TaihuLight, collecting per‑job traces from compute, forwarding, storage and metadata servers and aggregating ~100 GB raw/day with aggressive online+offline compression into about 10 TB over 18 months. It links jobs via scheduler and Lustre mappings, runs on Elasticsearch+Redis, detects modes (N-to-1, N-to-M), analyzes 116,000 jobs ≥32 nodes, exposes cache‑thrash and mapping over‑provisioning, drives fixes, and measures added cost at <1% overhead while operating at production scale on TaihuLight.

AI-written summary · about

chapters (12)

  1. 0:00Introduction
  2. 0:24Supercomputer IO
  3. 1:57Backend IO
  4. 3:04Beacon II
  5. 4:25Cluster Architecture
  6. 6:05Beacon System
  7. 8:49IO Performance Analysis
  8. 11:14IO Modes
  9. 12:56IO Volume Comparison
  10. 16:22Hidden Problem
  11. 17:44Performance Overhead
  12. 18:56Conclusion

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.