Cancer Genomics - a biologist and a developer
summary
Two concrete pipeline case studies: a serverless search called GT-Scan2 built on S3, DynamoDB, API Gateway, Lambda and SNS that runs CRISPR target queries at scale; and a large-scale ML pipeline using VariantSpark on Apache Spark (run via Databricks) that claims to analyze 3000 samples 80M features <30 minutes by massively parallel random forests. Benchmarks report ~80% speedup vs ADAM and ~90% vs R/Python; recommended deployment is Databricks SaaS with Jupyter notebooks and EC2 spot instances to cut multi-day jobs (55h) down to minutes.
AI-written summary · about
chapters (11)
- 0:00Cancer genomics pipelines
- 3:10Genomic data challenge
- 8:31GT scan 2 for CRISPR
- 14:30Genome-wide association studies
- 17:58VariantSpark and Hadoop
- 21:51Open-source libraries
- 27:12Machine learning in pipelines
- 34:51VariantSpark for gene discovery
- 42:17Deep learning optimization
- 46:08SaaS for genomic research
- 49:43Q&A
related talks
42mHow Novel Compute Technology Transforms Life Science Research
46mKeynote: Life Science Research at Scale
45mDigital Disruption in health & medical research
25mScaling Up Genomics with Spark
10mGenesis Final Round - Drylab AI
46mScalable Genomics with ADAM
04mMarius Eriksen Interview
12mHow data science can make your genome actionable