Streamlining the Cosmos: Pythonic Workflow Management for Astronomical Analysis
summary
Astronomical pipelines scale from single-galaxy processing to orchestration across HPC clusters, using Snakemake to auto-generate job submissions (no hand‑written SLURM) and to track per‑rule environments and dependencies; applied to James Webb pure‑parallel spectroscopic cubes (3D spatial + ~2000‑channel wavelength traces) and to surveys anticipating ~500 PB of raw Rubin data. Pixie (project‑scoped environments, Pixie v0.4) replaces slow Conda workflows for reproducible, fast Python environments; example pipelines and GitHub starter projects demonstrate the setup used.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:53Astronomical big data era
- 4:41Data processing challenges
- 8:15Snakemake for workflows
- 12:04Case study: JWST data
- 15:54Three-dimensional light data
- 19:43Complex workflow orchestration
- 20:46Reproducibility with Pixie
- 23:33Key takeaways
- 25:25Q&A
related talks
34mData pipelines powered by Open source and fun
23mA DIY Guide to Building Data Pipelines with Python
11mProcessing TBs of Astronomy Data at Scale with the AC3 Horizon Research Project
45mFrom telescope to data centre: adventures in astronomical data pipelines
07mCloud architecture for population-scale analyses of genomes and AI.
01h 11mProgramming for the planet
43mData Pipelines à la Mode
01h 04mBuilding Efficient and Frugal Image Data Processing Pipelines