The IceCube data pipeline from the South Pole to publication
summary
A working pipeline moves ~1 TB of raw IceCube data per day from the South Pole, filters most of it on-site to fit a ~100 GB/day satellite link, and produces IceTray frames (I3 files) that serialize event dictionaries for downstream work. The stack mixes Python C++ Java and FORTRAN, uses a tableio layer to hammer frames into rows, and writes to HDF5 and ROOT backends for analysis. A published result reduces billions of events to 28 high‑energy candidates and achieves ~5σ significance (≈1 in 22 million).
AI-written summary · about
chapters (17)
- 0:00IceCube introduction
- 1:51What is a neutrino
- 7:10South Pole detection
- 9:37How IceCube works
- 11:00IceCube data pipeline
- 16:58IceTray framework
- 22:57Filtering South Pole data
- 24:22Simulating data grid
- 25:28Data to tabular formats
- 26:28Data analysis histograms
- 28:37Astrophysical neutrinos case
- 30:45Summary Python usage
- 31:19Q&A ISP fluctuations
- 32:02Q&A Noise methodology
- 33:05Q&A South Pole environment
- 34:30Q&A Data uncertainty
- 35:53Q&A Open science release
related talks
44mRadio Astronomy Reduction Pipelines Using Python and Various Container Technologies
30mOn the Fly and On the Flight: Scientific Data Analysis Beyond the Beamline
31mOpenDataCube - Petabyte Scale Satellite Imagery Processing in Python
28mBridging Python and Apache Iceberg: The Power of PyIceberg
42mMaking Sense Out of Flight Test Data with Python
32m7 steps to build-your-own data pipeline - for day 1 of your startup
30mSoftware for the Murchison Widefield Array radio telescope
16mThe Open Data Cube in a Box