Experiences in Managing the Performance and Reliability of a Large-Scale Genomics...
summary
GDC (Genomic Data Commons) pipelines on KVM VMs age over days, producing rising EPT violations that correlate with multi× slowdown; repeated runs (five alignment jobs, six vCPUs each) across >10 days show violation counts and job times steadily increase. Two operational detectors are built: host-level tracing with an address-distance distribution metric (low overhead, early warning) and a simple vCPU efficiency ratio (no overhead, detects only after slowdown). Memory defragmentation reduces violations but never restores fresh-VM performance; AWS/Google VMs run more consistently at higher cost.
AI-written summary · about
chapters (4)
related talks
11mEnsuring Reliability in Healthcare Data Engineering
25mTackling operational and capacity utilization challenges at a large scale enterprise deployment
07mCloud architecture for population-scale analyses of genomes and AI.
47mCloud Native and Sustainable, Reproducible Scientific Computing
56mLarge-Scale Distributed Processing in Google Cloud - 7 lessons...
36mIntroduction to High Performance Computing on Google Cloud Platform
16mUnlocking unallocated cloud capacity for long, uninterruptible workloads
26mLessons Learned: Optimizing KVM Performance for EHR Systems