Stop Burning Tokens: Why self-improvement needs domain expertise first
summary
A minimal self‑optimization loop on arXiv paper classification runs on Langfuse with a small base LLM as classifier and a frontier optimizer proposing prompt edits via cloud code, using a 200/100/300 split and stopping at 15 runs or 92% accuracy. Error‑cluster analysis yields error‑cluster driven prompt updates that add rules and examples, lifting accuracy from 68% to about 83% — a 15% improvement that generalizes to 80.2% on an unseen test set.
AI-written summary · about
chapters (12)
- 0:00Introduction and the goal of avoiding token waste
- 0:24The current trend of "designing loops" vs. prompt engineering
- 0:47Why coding (with its clear "compile" target) set a false precedent
- 1:20Challenges of defining target functions in non-coding domains
- 2:30Case study: The arXiv paper classification experiment
- 3:30Setup of the minimal self-optimization loop
- 5:48The step-by-step iteration process and stopping criteria
- 7:00Results: Achieving a 15% improvement in accuracy
- 9:53Deep dive into the reasoning behind the improvement
- 11:26Translating binary "high signal" feedback to other applications
- 13:12Defining what "good" looks like for your specific domain
- 15:05The importance of human-agent collaboration and data review
related talks
29mLocal LLM fine-tuning: a practical example
52mBuild a Prompt Learning Loop
49mBen Hall & Barbara Fusinska
20mEverything I Learned Training Frontier Small Models
37mOptimal on Paper, Broken in Reality
25mWeights vs. Words in Fine-Tuning & Enhancing Agentic AI Workflows
21mFrom Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
52mReduce LLM calls with Vector Search Design Patterns