Semantic Blindness: 500,000 Sensors Confused an LLM
summary
Claims a failure mode called Topology Trap and two others — sensors-as-strings confuse embeddings, massive context causes attention collapse, and near-identical tags trigger a Repetition Kill Switch that stops output. Engineers feed 500,000 sensor names to LLMs, see correctness drop from ~80% at small scale to ~30% at 460,000, and token burn of 116 million per validation. Built a hierarchical linearizer plus planner-to-deterministic-resolver pipeline that yields 100% accuracy in 66 real cases and cuts tokens to ~390k.
AI-written summary · about
chapters (16)
- 0:00Semantic blindness problem
- 1:03Naming chaos breaks LLM systems
- 2:27RAG and naive LLM fail at scale
- 3:30Sharded LLM solutions for naming
- 4:32Hierarchical structure solves scalability
- 5:35Linearizer for equipment paths
- 6:58LLMs plan but struggle to search
- 8:02Pre-indexed trees for set operations
- 9:06Pattern matching for vague queries
- 10:09End-to-end query processing
- 10:51Production results and evaluation
- 12:40Flat cost scaling for data centers
- 13:42LLM vs deterministic code balance
- 15:29Maturing AI systems for reliability
- 15:50Key takeaways and next steps
- 16:11Q&A
related talks
34mTerrible tokenizer troubles in large language models
15mStop Stuffing Your LLM's Context Window
35mReading the Mind of an LLM
39mHallucination-Free LLMs: Strategies for Monitoring and Mitigation
49mBen Hall & Barbara Fusinska
46mThings you never dared to ask about LLMs
46mUnder the Hood of LLMs — all those questions you never dared ask about them
55mLessons Learned From Implementing Generative AI Solutions