Training and Optimisation of Large Transformer Models: An ATLAS and CERN Use Case - Ricardo Rocha, CERN & Maxence Draguet, University of Oxford
Talkat Cloud Native AI Day Europe 2024 · on Tuesday, 2024-03-19· at 09:15· Pavilion 7 | Level 7.3 | Room S03·
summary
CERN runs a Gen2 Transformer for ATLAS jet flavour tagging on a Kubeflow CERN ML cloud, using Kubeflow + Katib for distributed training and hyperparameter search and Gen2 Transformer as the model. Adopting Maximal Update Parametrization (muP) lets hyperparameter searches run on a small model and transfer to the full model, cutting wall‑time cost (full width 256 = 2.3M params, ~140 min/epoch on two GPUs; small model ~20 min/epoch on one GPU, ~4× hyperparameter coverage per equal compute). GPU sharing uses MIG + GPU Operator and PCI passthrough; multicluster scheduling and advanced co‑scheduling remain unresolved.
AI-written summary · about
chapters (8)
- 0:00CERN and ATLAS intro
- 4:15CERN’s cloud-native journey
- 7:09GPU-based ML challenges
- 13:32Hybrid cloud scheduling needs
- 15:19Higgs boson decay classification
- 23:08Maximally parameterized networks
- 31:40KubeFlow with MUP for HPO
- 33:11Q&A on Kubernetes and GPU sharing
related talks
29mA Scalable Platform for Training and Inference Using Kubeflow
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
16mGalvatron: An Automatic Distributed Training System for Efficient Large...
24mJet Energy Corrections with GNN Regression using Kubeflow @ CERN
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
34mTransformers End to End Experience training...
31mBuilding Large-scale Localized Language Models
16mScaling Massive Transformer Training with KubeRay