Simplicity For Scale- Analyzing 15 Million DNA Samples With Python
summary
Pipeline ingests 15 million DNA tests and replaces a year‑long Hadoop/Java batch system with event‑driven Python using Celery plus a C++ germline executable, shrinking ~125k lines of Java to ~25k Python/C++ and turning a matching job from ~6.5 hours into ~6 minutes (about a 64x speedup). The team trades scheduled batches for per‑sample tasks, keeps the 24‑hour processing window, and proves simpler tooling plus native code for hotspots actually scales.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 4:02Biological data pipelines
- 7:14Original DNA pipeline issues
- 12:29Hadoop for DNA matching
- 14:57Hadoop pipeline problems
- 23:21Python over Hadoop choice
- 25:09Celery event-driven processing
- 30:08Optimizing matching with C++
- 35:47Migration lessons learned
- 38:35Simplicity as solution
- 40:22Q&A
related talks
42mHow Novel Compute Technology Transforms Life Science Research
50mCancer Genomics - a biologist and a developer
47mPython for Bioinformatics
01h 01mParallel Python Analyzing Large Data Sets
25mScaling Up Genomics with Spark
38mPython in Bioinformatics
14mBig data, AI, the genome, and everything
23mUsing Python for the PCR Design