Is your LLM any good at writing? Benchmarking on creative writing and editing tasks
summary
A systematic benchmark tests 18 models on 15 concrete writing tasks spanning date & greeting formatting, following and avoiding explicit style instructions, vocabulary and syntactic diversity, and self-evaluation. Every model nails date and greeting formatting; conciseness is 99% for reasoning-focused models like O3-mini; the blacklisted words test stumps all models (none exceed ~86%). Larger models do not always win; O3-mini tops instruction adherence while Claude 3.5 Sonnet trades better balance with higher syntactic diversity.
AI-written summary · about
chapters (8)
- 0:00LLM writing usage and bias
- 3:23LLM writing style problems
- 7:57Benchmark for writing tasks
- 13:38Benchmark results and performance
- 18:13Diversity in LLM content
- 19:17Why LLMs lack diversity
- 21:42Benchmark sharing and future work
- 22:04Q&A
related talks
35mEvaluating LLM systems: the nuts and bolts
14mTailor-Made LLM Evaluations: How to Create Custom Evaluations for Your LLM
32mA Practical Guide to LLM Evaluation
47mValidating LLMs – A True Story
01h 18mThe Enigma of LLMs: on Creativity, Compositionality, Pluralism, and Paradoxes
30mLLM Query Benchmarks: Cypher vs SQL
10mWhat goes into the making of LLMs
16mBest Practices for Open Multilingual LLM Evaluation