The Miranda Hypothesis: How Hamilton Poisoned Persona Evals
summary
Claims that RLHF amplifies a corpus-driven failure mode called Miranda distortion and that standard persona evals miss it. Shows a reproducible instrument, the Prism Experiment: four Lincoln moments, three seeding conditions (bare, biography, primary), five historian-written questions, and a weighted rubric with Anachronism Detection 40%, Documentary Consistency 35%, Contextual Plausibility 25%. Demonstrates context-window anchoring, expert-authored gold sets, and concrete pre-registration as the practical eval gate that catches anachronistic compositing.
AI-written summary · about
chapters (13)
- 0:00Role-playing language agents intro
- 3:41Eval measurements core question
- 6:33Persona evaluation paradigms
- 10:08Convincingness vs fidelity
- 16:02Miranda hypothesis explained
- 22:06Cognitive to epistemic simulation
- 27:10Role-playing systems architecture
- 33:38PRISM evaluation instrument
- 42:23Lincoln experiment pre-registration
- 47:42Voice not valid evaluation
- 51:43Domain experts in evaluation
- 54:53Language models in archive
- 57:01Conclusion and call to action
related talks
26mReducing Hallucinations and Evaluating LLMs for Production
33mThe Silent Crash:Why Your RAG Evaluation Metrics Are Lying to You
11mThe Unbearable Lightness of (Agentic) Evaluations
46mEnhancing Agent Reliability Through Evaluation-Driven Development
56mMirror, mirror: LLMs and the illusion of humanity
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
15mReducing Hallucinations in Conversational Systems
32mWhy do LLMs hallucinate?