MeerKAT Astronomy Data Store Deployment and Operations
summary
MeerKAT runs two Ceph clusters — a Cape Town archive with 2640 hard drives and 55 NVMe devices and an on-site Karoo cluster with 240 hard drives and 480 SSDs — built from bespoke low‑RFI storage pods (48x3.5" drives, one NVMe, boot SSD, 128 GB RAM, single Xeon, 25 GbE). Correlator output reduces raw streams to a few terabytes per multi‑hour observation. Data movement uses a custom transfer daemon over an unreliable link; deployment and upkeep are automated with Ansible and MAAS, monitored by Prometheus/Grafana and Netdata.
AI-written summary · about
chapters (9)
- 0:00Introduction and talk overview
- 7:36History of astronomy data storage
- 10:50MeerKAT telescope overview
- 17:19MeerKAT data flow and SAFE usage
- 19:13Custom storage pod development
- 26:46Processing and storage infrastructure
- 29:17Automation and deployment tools
- 35:48Monitoring and observability tools
- 40:51Community and future plans
related talks
20mKeynote Welcome
27mLessons learnt expanding Cambridge University's CSD3 Supercomputer with OpenStack
19m10 years of OpenStack at CERN (From 0 to 300k cores)
42mDrupal, Kubernetes, Production
15mThe CERN Cloud Infrastructure: 5 years in 15 minutes
46mDigitalizing a critical, highly distributed infrastructure company
18mScaling Bare Metal Provisioning with Nova and Ironic at CERN
16mTips and Tricks for Managing and Administering Ceph Clusters