How Novel Compute Technology Transforms Life Science Research
summary
Team describes genome‑scale pipelines that use serverless (AWS Lambda) and Apache Spark / VariantSpark to run feature selection across 22,000 samples (~1.7 trillion comparisons), identifies the four synthetic “hipster” loci in a demo, and reports per‑run costs like AU$200 (1M features) to AU$8,000 (50M features, 10k samples). CRISPR guide search (CTScan) moves from minutes to seconds, enabling scalable API access via DataBricks and SageMaker and aiming to improve 7/10 embryo edit success rates.
AI-written summary · about
chapters (23)
- 0:00AGILE INDIA 2018
- 0:45Overview
- 2:14Australia's innovation catalyst
- 3:39Genome holds Blueprint for Every Cell
- 3:59Genomics affects looks, disease risk, and behavior
- 5:53Finding disease gene(s)
- 7:31Machine learning on 1.7 Trillion data points
- 8:26Population-scale genomic data analysis requires BigData solutions
- 10:01Wide Random-forest scalable with features and samples
- 11:12What business application require 'wide data?
- 13:06Wide-random forest vs "traditional parallelization"
- 15:38Typical workflow (cloud pipeline pattern)
- 17:24Option for building the pipeline
- 19:45Demo: VariantSpark
- 26:35Building a community
- 27:20Cloud Performance Comparison
- 28:49Releasing a Python API
- 31:24Improve performance
- 32:18First serverless research
- 37:18Cloud pipeline pattern: wide ML
- 38:52Cloud pipeline pattern: serverless web service
- 40:21Three things to remember
- 41:47Transformational Bioinformatics
related talks
50mCancer Genomics - a biologist and a developer
45mDigital Disruption in health & medical research
46mKeynote: Life Science Research at Scale
14mBig data, AI, the genome, and everything
25mScaling Up Genomics with Spark
51mSimplicity For Scale- Analyzing 15 Million DNA Samples With Python
23mGenomics and AI on Cloud Foundry
10mGenesis Final Round - Drylab AI