Cloud architecture for population-scale analyses of genomes and AI.
summary
Hybrid architecture for population-scale cancer genomics combines local supercomputing (Lustre + Singularity) for heavy Nextflow pipelines and on‑prem Docker/MongoDB for postprocessing with an AWS layer (S3, EC2 spot, AWS Batch) for burst compute. Pipelines process ~200 billion bases per sample, runtimes hours–days, and scale to hundreds‑to‑thousands of genomes. High upstream failure rates force per‑repository QC pipelines and an Airflow orchestration layer that lowers reruns. A Parquet/Delta Lake/Spark store with Glow fixes ad hoc file access. ML training remains migrating to cloud.
AI-written summary · about
related talks
47mCloud Native and Sustainable, Reproducible Scientific Computing
27mVisualizing Cloud Systems
38mCloud Science
31mReliability and Resilience in the Cloud
28mHerding Data to the Cloud
44mBest practices for analytics and generative AI on AWS(ANT302)
43mA Year of Diagnosing Pediatric Cancer with Event-driven Architecture
27mFrom Supercomputing to Serving: A Case Study Delivering Cloud Native Foundation Models