Scalable Genomics with ADAM
summary
ADAM is a schema‑first storage and processing stack built on Apache Spark and Parquet that replaces bulky VCF text files with typed columnar rows, reducing storage and enabling distributed reads. The pipeline converts VCF to ADAM, compresses data, runs k‑means on genotype feature vectors for 250 samples (chromosome 6 slice: 1.5M bases → ~20,000 variants), and scales from 4 to 40 nodes with task times dropping from minutes to seconds claiming up to a 150× speedup.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:16What genomics needs
- 7:00Traditional data format issues
- 8:05Parquet for genomics storage
- 14:53ADAM and Parquet
- 17:49Genomics data standards
- 20:02ADAM API with Spark
- 22:091000 Genomes case study
- 29:04VCF to ADAM conversion
- 31:14k-means on genomic data
- 36:06Scaling on AWS
- 39:07Key findings and future
- 40:57Acknowledgments
- 41:20Q&A
related talks
21mInteractive Machine Learning on Genomics Data with Spark Notebook
25mScaling Up Genomics with Spark
50mCancer Genomics - a biologist and a developer
46mKeynote: Life Science Research at Scale
42mHow Novel Compute Technology Transforms Life Science Research
26mBuilding a scalable genomic matching algorithm
01h 08mGamgee: A C++14 library for genomics data processing and analysis
08mQ n A with Dr Denis Bauer