A Generalization Bound for Online Variational Inference
summary
Proposes two sequential variational updates called SVA and SVB and proves regret bounds matching Bayesian online guarantees under convexity and strong-convexity assumptions; derives explicit Gaussian mean/variance updates and shows SVA analysis extends to general location–scale families while SVB is proved only for Gaussians, giving √T regret in convex case and log T in strongly convex case. Empirical comparison across many datasets finds online gradient worst, SVA/SVB competitive, and NGVI consistently best despite lacking the paper’s convexity-based theory.
AI-written summary · about
chapters (4)
- 0:00Start
- 1:45Motivation
- 3:57Bayesian inference and variational approximations (Generalized) Bayesian inference
- 14:12Online gradient algorithm (OGA)
related talks
33mStochastic Optimization for Big Data Machine Learning Problems
01h 01mRobustness, Stochastics, Uncertainty 2
01h 12mStochastic Second Order Optimization Methods II
29mImplicit Regularization in Nonconvex Statistical Estimation
42mBypassing the Impossibility of Online Learning Thresholds: Unbounded Losses and Transductive Priors
21mProbabilistic Programming with Programmable Variational Inference
45mStudying Generalization in Deep Learning via PAC-Bayes
09mConvex Programs, Computational Phase Transitions, and Robust Algorithms