"Four years of Datomic powered ETL in anger with CANDEL"
summary
Four years of production use show CANDEL handles multi-omic and clinical harmonization at scale, powering peer‑reviewed papers in Nature and Cell. The team builds an ETL compiler called Pret that converts EDN mappings into Datomic transactions, a file manager raw sugar, and an image feature tool mantis; the system ingests tens of thousands of patients across dozens of assays, lets schema changes propagate without code rewrites, and successfully enabled new biomarker discoveries.
AI-written summary · about
chapters (11)
- 0:00CANDLE project introduction
- 3:50Motivation for CANDLE
- 10:41Biological data integration challenges
- 14:34Prep automates ETL with Datomic
- 21:17CANDLE database tools
- 27:34Data science workflows with CANDLE
- 30:47Research projects using CANDLE
- 32:59Lessons learned and improvements
- 37:33Scaling large datasets
- 39:15Open sourcing CANDLE
- 40:20Project status
related talks
37m11 insights after 11 years with the functional database Datomic
42mFour Years of Data Mesh in Practice
36mSimplifying ETL with Clojure and Datomic
42mYet & Datomic Immutable Facts Mutated Our Stack
40mThe power of Datomic database
31mWormBase database migration to Datomic on AWS: A case Study
35mIt's Just Data
37m11 insights after 11 years with the functional database Datomic