Domain Adaptation and Related Areas 2024
2024-11-12..2024-11-15
Berkeley, United States
who it's for
research scientists, NLP researchers, applied machine learning engineers, theoretical machine learning researchers, privacy and federated engineers, healthcare data scientists, computational ecologists and environmental ML practitioners
topics
label noise in classification; pre-trained language models; pseudo-labeling under covariate shift; transformers learning causal structure; weak-to-strong generalization via LLM refinement; nonparametric local convergence rates; user-level federated local differential privacy; discrepancy-based domain adaptation; distribution shift in ecological vision; best-of-n sampling distributions; adaptive data collection via autoregressive generation; open-source foundation model practices
| title | author | length |
|---|---|---|
| Label Noise: Ignorance is Bliss | Clayton Scott | 42m |
Defines relative signal strength as a pointwise measure of transferability between noisy and clean posteriors and proves matching upper and lower excess-risk… | ||
| The Evolution of Domain Transfer in the Era of Pre-trained Language Models | Robin Jia | 43m |
Pre-training changes what models find hard: short tests with distracting sentences expose sensitivity failures that are largely fixed by data augmentation of… | ||
| The Data Addition Dilemma | Irene Y Chen | 37m |
Combining datasets sometimes decreases target performance: experiments on the 208-hospital eICU corpus using an LSTM to predict 24-hour mortality show adding… | ||
| How Transformers Learn Causal Structure with Gradient Descent | Jason Lee | 38m |
Gradient descent on a two-layer disentangled transformer with one-hot tokens and softmax attention provably encodes a latent DAG: the first attention layer… | ||
| Pseudo-Labeling for Covariate Shift Adaptation | Kaizheng Wang | 39m |
Method splits labeled source into two parts to produce a set of candidate regressors and an independent imputation model with low bias, uses that imputer to… | ||
| Transfer learning for weak-to-strong generalization | Yuekai Sun | 42m |
Claim: weak-to-strong generalization is possible by eliciting latent knowledge from pre‑trained LLMs. Method: apply refinement via in‑context learning—use weak… | ||
| Transfer learning via local convergence rates of the nonparametric least squares estimator | Johannes Schmidt-Hieber | 41m |
Shows that the empirical risk minimizer over 1‑Lipschitz functions attains minimax local rates under covariate shift by proving a pointwise bound tn defined… | ||
| User-level and federated local differential privacy | Yi Yu | 37m |
Shows that imposing user-level LDP (each user holds T samples) interpolates between item-level LDP and central DP and changes minimax rates: central mean error… | ||
| A Discrepancy-Based Theory of Adaptation | Mehryar Mohri | 54m |
Introduces labeled and unlabeled discrepancy as the central divergence, derives finite-sample estimation via Rademacher complexity and gives explicit bounds… | ||
| Distribution shift in ecological data: generalization vs. specialization, | Sara Beery | 41m |
Ecological vision faces concrete tradeoffs: global models improve aggregate accuracy but fail local decisions, while specialized local models or fine-tuning… | ||
| On Spurious Associations and LLM Alignment | Victor Veitch | 45m |
Claims an analytic characterization of the KL-regularized objective and shows the best-of-n sampling distribution is essentially Pareto-optimal for the win… | ||
| Talk by Zachary Lipton | Zachary Lipton | 43m |
Argues that academic distribution-shift theory rarely helps in practice and shows a concrete counterexample: a deployed audio→note pipeline that strings… | ||
| Causally motivated robustness to shortcut learning | Maggie Makar | 45m |
Claims that causally-motivated regularization—combining importance weighting to mimic an “ideal” uncorrelated distribution with an MMD penalty that enforces… | ||
| Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains or the Whole Domain | Tengyu Ma | 42m |
Shows a negative result that wide neural nets and RKHS contain arbitrarily spiky functions, so multiplicative extrapolation can be infinite unless the model… | ||
| Language-guided Adaptation | Trevor Darrrell | 52m |
Language acts as explicit advice to fix domain shift: adding an attention loss from Grad-CAM/RISE to force models to be right for the right reasons; using CLIP… | ||
| Adaptive Data Collection via Autoregressive Generation | Hongseok Namkoong | 45m |
A framework trains an autoregressive sequence model on massive historical interaction logs to learn a prior over content, then generates autoregressive… | ||
| Scalably Understanding AI with AI | Jacob Steinhardt | 46m |
Trains investigator agents with supervised fine-tuning and DPO to invert outputs into prompts, then applies them to jailbreaks and hallucination searches… | ||
| Open-Source and Science in the Era of Foundation Models | Percy Liang | 44m |
Argues open access matters and backs it with concrete results: a position paper finds only 9 of 30 model reports disclose train/test overlap; builds… | ||
| Domain Adaptation-- 20 years of theory chasing practice | Shai Ben-David | 37m |
Describes a 2005 NLP fix that represents each token by a pivot-word window embedding and then trains a linear classifier on Wall Street Journal labels that… | ||
| Panel Discussion | Rich Zemel, Shai Ben-David, Kamalika Chaudhuri, Percy Liang, Hongseok Namkoong | 49m |
Panelists debate whether scaling plus architecture wins: larger pretrained models (CLIP/LLMs) empirically fix many classic domain-adaptation traps (e.g… | ||
| TBD | Kamalika Chaudhuri | 45m |
Proposes a statistical IRM scaffold: three concrete conditions—source-preservation, contraction, Bayes-unifies—and proves that with an oracle representation… | ||
| nothing matches | ||