Scalable Multi-Language Data Analysis on Beam: The Cuneiform Experience
summary
Cuneiform is a minimal workflow language implemented on the Erlang VM that automatically derives task- and data-parallelism from functional scripts and runs real bioinformatics pipelines (variant calling, methylation, RNA-seq) by calling Perl Python R Bash tools through Erlang ports. The system uses a cache and scheduler, a Petri-net execution model and an Petri net OTP library (Apache 2), and integrates with job systems so it runs on HDCondor and Hadoop YARN via highway; the prototype is small and working.
AI-written summary · about
chapters (9)
- 0:00Start
- 1:31Scientific Workflow System
- 8:39Foreign Language Interfacing
- 19:02How Do You Program Model a Program in English
- 22:09Operational Semantics
- 25:17Execution Environment
- 31:12Reasons Why Large-Scale Data Analysis Is Hard
- 34:21Is There a Hadoop Library
- 35:33Interface to Other Languages
related talks
38mCuneiform: A Functional Workflow Language Implementation in Erlang
10mDemocratizing the pipeline with Apache Beam and Dataflow
52mApache Beam, a unified model for Big Data processing
51mSimplicity For Scale- Analyzing 15 Million DNA Samples With Python
41mPython, Java, or Go: It's Your Choice ...
47mAdvances in Stream Analytics: Apache Beam and Google Cloud Dataflow deep-dive
04mMarius Eriksen Interview
36mMachine Learning with Apache Beam