Judgement Day: Benchmarking "Black Box" LLMs With Open Legal Datasets
summary
A controlled experiment evaluates GPT-5, Gemini 2.5 Pro and DeepSeek‑R‑One on LegalBench using a custom async Python harness that runs 4,302 reasoning traces and enforces strict JSON outputs. Injecting partial statutory retrieval produces statutory myopia: overall accuracy drops by 9 percentage points, hearsay tasks fall 21 points, and per‑model losses range 6–9 points. The pipeline includes a blacklist control and suggests graph‑structured KBs and completeness checks before injection.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:05Statutory myopia in RAG
- 2:28Black box LLMs in law
- 3:55Statutory myopia matters
- 5:19RAG experiment design
- 8:10Evaluation harness design
- 10:56Task and model selection
- 13:48Retrieval control protocol
- 15:55Personas impact performance
- 17:00Results with JSON matching
- 18:04RAG can worsen models
- 19:27Myopia affects all models
- 20:51Domain-specific results
- 22:17Real case of statutory myopia
- 24:04Mechanism behind myopia
- 25:51Solutions for RAG pipelines
- 27:15Verify before injecting context
- 28:21Limitations and future work
- 29:05Three key takeaways
- 30:13Q&A
related talks
24mHarnessing the Power of LLMs in Advanced Analytics
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
40mLLM for Coding, the State and lnitiative
40mTechniques to avoid common LLM pitfalls
25mWhy LLMs Still Fail: Reasoning, Robustness, and Safety
24mHow to Keep Your LLM Chatbots Real - A Metrics Survival Guide
31mBeyond LLMs and Maturing RAG use cases
35mEvaluating LLM systems: the nuts and bolts