Text2Cypher Beyond English: Initial Evaluation with Foundational LLMs
summary
Paper-style evaluation of Text2Cypher using foundational LLMs compares English, Spanish and Turkish by feeding the same Cypher queries but translated questions only and measuring both text similarity and database results with execution exact match. Three foundational models (including 3.1 and 2.7 family releases) run on a released Hugging Face test set; English scores highest, Spanish drops moderately, Turkish degrades substantially, and translating prompts yields little improvement — schema remaining in English likely compounds failures.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 0:46Text2Cypher and databases
- 3:39Why support other languages
- 4:45Evaluation setup
- 7:13Dataset preparation
- 8:20Models and metrics
- 9:50Evaluation results
- 12:02Conclusion and future work
- 12:45Q&A
related talks
14mBuilding a Text2cypher Model
30mLLM Query Benchmarks: Cypher vs SQL
16mBest Practices for Open Multilingual LLM Evaluation
44mKeynote: The Art of Language: Mastering Multilingual Challenges in LLMs
44mKeynote: The Art of Language: Mastering Multilingual Challenges in LLMs
57mMachine Translation of Human Languages in the Age of LLMs: Is This the End of the Language Barrier?
22mBuilding Cypher Queries With Open-Source LLMs
24mHarnessing the Power of LLMs in Advanced Analytics