Processing TBs of Astronomy Data at Scale with the AC3 Horizon Research Project
summary
AC3 automates end to end processing of telescope data by containerising tools like Starlight and selecting PPXF or Steckmap as appropriate then deploying them across cloud and edge with an AI driven lifecycle manager and network operator. Raw terabyte observations are split into ~100 MB chunks, moved via EDC data connectors (Ionos extended S3 bidirectional connectors), queued and processed with horizontal and vertical autoscaling based on CPU, RAM, queue length and job time. Tested with Roque de los Muchachos datasets in collaboration with University of Madrid and running in production-like pipelines.
AI-written summary · about
chapters (9)
- 0:00AC3 Horizon Project intro
- 1:25AC3 platform use cases
- 1:46Focus on astronomy data
- 3:56Astronomy data workflow
- 6:06Secure data transfer with EDC
- 7:52Monitoring and autoscaling
- 9:07Data integrity Q&A
- 10:11AI-driven networking in AC3
- 11:20Closing remarks
related talks
45mFrom telescope to data centre: adventures in astronomical data pipelines
36mRapid Processing of NASA Satellite Data Stored with Ceph
06mLightning Talk: SKAO: Next-Generation Radio Astronomy-Driven Big Data
05mAlex Kudriashova Interview
51mWhen the Universe Calls
35mStreamlining the Cosmos: Pythonic Workflow Management for Astronomical Analysis
20mFull Lifecycle Data Analysis on a Large-scale and Leadership Supercomputer...
28mHerding Data to the Cloud