Adaptive Generalization: The Role of Dynamic Evaluation in Low-Resource Settings
summary
Introduces a adaptive reasoning graph that converts benchmark examples into graphs, perturbs depth/width and regenerates text to test memorization and robustness; math (GSM8K) accuracy falls from ~0.8 to ~0.4 under perturbations and social-reasoning refusal and bias rates rise. Builds a crowd-sourced culture bank (TikTok-derived, ~730 groups) and shows fine-tuning raises cultural entailment and model wins versus vanilla models. Prototypes rehearsal systems for counselor training using constitutional constraints and expert-tuned mentors, deploying in medical courses with preliminary learner-skill gains.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 4:57Dynamic evaluation of LLMs
- 24:46Cultural awareness in LMs
- 47:58Social skill training with AI
- 1:01:00Q&A
related talks
32mScaling Test-time Inference Compute & Rise of Reasoning Models
39mUnlocking reasoning in Large language models
24mThe Rise of Reasoning Models
31mMaking the most of test-time compute in LLMs
43mDiscussion
42mReasonable AI
22mAutomated rationale generation: a technique for explainable AI and its effects on human perceptions
01h 16mGeneralization in natural language processing