In-Context Principle Learning from Mistakes
summary
Claims that extracting brief, generalizable rules from contrasts between sampled wrong solutions and ground-truth answers improves few-shot reasoning. Method samples multiple chain-of-thought answers from an LLM for a few labeled examples, filters incorrect ones, prompts the model to write low-level lessons, optionally summarizes them into high-level rules, and prepends those to standard few-shot prompts. Works with GPT‑4/Gemini/GPT‑3.5 (open models like Llama‑2 fail), costs about 30 LLM queries, and yields gains (e.g., 7.5% on DROP, 3.3% HotpotQA, ~1% GSM8K/MATH) but is inconsistent across datasets.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:17Calculus teaching motivation
- 3:06Few-shot chain-of-thought prompting
- 4:58Negative examples in in-context learning
- 8:50Including negative examples
- 12:22Self-refine limitations
- 19:13Introducing LEAP
- 26:44Generating learned principles
- 28:42Experimental results
- 31:37Learned low-level principles
- 37:10Case studies
- 41:26Benchmark-specific principles
- 43:55Principle selection
- 45:19Cross-model transferability
- 46:29Open-source model testing
- 49:09Reflection and self-correction
- 50:57Summary
- 52:00Q&A
related talks
26mLLM Inference with Few-Shot Prompting and Chain of Thought
26mFew-Shot Learning in Conversational AI
39mUnlocking reasoning in Large language models
31mBeyond LLMs and Maturing RAG use cases
25mWhy LLMs Still Fail: Reasoning, Robustness, and Safety
27mExploring the Mechanisms of LLM Reasoning Capabilities
29mBeyond Solo: Enhancing LLMs through Collaborative Dialogue with GAN-Inspired Approaches
01h 39mMastering LLMs: From Prompt Engineering to Agentic AI