Scale Learning and Reasoning Across Heterogeneous Gradients and Semantics
summary
Proposes Bi^2Clip, a coordinate-wise bi‑directional clipping optimizer that approximates adaptive preconditioning without storing per‑client statistics (two clipping thresholds, SGD‑level memory) and provably handles heavy‑tailed gradient noise (alpha‑moment setting) while matching Adam‑like performance on GLUE, CIFAR‑100 ViT and SQuAD/Qwen2.5. Adds staleness‑aware aggregation that downweights stale updates to improve asynchronous convergence. Introduces federation‑over‑text where clients share metacognitive summaries to iteratively build an insight library (tested across ~8 math benchmarks and ~20–30 paper subareas), reducing token generation and improving accuracy.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:30Heterogeneous gradient challenges
- 4:34Heavy-tailed noise in gradients
- 7:21BiClip optimizer
- 10:03BiClip in federated optimization
- 12:15Convergence analysis and results
- 15:47BiClip performance on data
- 18:52Asynchronous training
- 21:19Clipping in async setups
- 23:39Staleness-aware aggregation
- 26:14Federated inference over semantics
- 29:25Federation: text vs gradients
- 32:03Math problem solving
- 35:55Scientific discovery applications
- 39:22Future directions
- 44:38Q&A
related talks
26mGeoClip: Geometry-Aware Clipping for Differentially Private SGD
17mApproximate Caching for Efficiently Serving Text-to-Image Diffusion Models
52mLanguage-guided Adaptation
01h 14mDistributed and concurrent optimization for machine learning
01h 05mTutorial: Federated Optimization, Part III
47mHow Intelligent Compute Accelerators (ICAs) are Shaping the Future of HPC & AI
33mTradeoffs in Large Scale Learning: Statistical Accuracy vs. Numerical Precision
52mSplitting Gradient Descent for Incremental Learning of Neural Architectures