Validation by Listening
summary
Claims a reference-free ET shuffling method can validate translators using only target text, tests it with GPT-4.1/3.5 on real and synthetic conlangs including constructed conlangs (Kelen) and model-made alien languages, finds the shuffle score correlates ~0.63 with standard ground-truth metrics (GEMBA/BLEURT). Tools: LLM translators, permutation-ranking by a fixed evaluator. Reports clear failure modes (repeated chunks, line-number or hallucinated templates) and a theoretical analysis showing when passive listening beats costly interactive grounding. Evaluation works modestly; training still needs grounding.
AI-written summary · about
chapters (11)
- 0:00Validation by listening
- 1:44Reference-free translation evaluation
- 3:51Shuffle target ET evaluation
- 10:19How shuffle target ET works
- 18:06Alien language testing
- 24:16ET versus ground truth correlation
- 27:44Failure modes of shuffle target ET
- 36:36Interaction improves evaluation
- 42:00Interactive learning benefits
- 49:47Listening outperforms interaction
- 54:05Conclusion and future
related talks
33mTranslation Augmented Generation - Breaking Language Barriers in RAG...
46mBeyond BLEU & ROUGE — Evaluating LLMs and AI Systems
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
57mArtificial Languages are Dead. Long Live Artificial Languages!
28mTrustworthy LLMs: Vibe checks are not all you need
01h 02mAI Effects on Research and Education: A Programming and Software Language Perspective
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications