Statically Contextualizing Large Language Models with Typed Holes
summary
Integration of a typed live editor with LLM completions produces fewer hallucinations by sending a Hazel language server–constructed static slice: inferred hole type, recursive type definitions, and relevant function headers. Implementation in Hazel plus a TypeScript port yields iterative LLM edits driven by type error messages. Evaluation on a new MVUBench dataset shows that including type definitions dramatically increases passing unit tests versus header-only or vector-retrieval baselines, while exhaustive retrieval matches accuracy at much higher token cost. ChatLSP is proposed to expose these capabilities via LSP.
AI-written summary · about
chapters (8)
- 0:00Typed hole contextualization
- 1:04Typed hole completion in Hazel
- 4:36Static contextualization problem
- 6:44Contextualization strategies
- 7:25Type-focused prompt construction
- 10:59Evaluation with MVU Bench
- 13:51Extending to other languages
- 15:40Q&A
related talks
53mSemantics-based program manipulation in the age of LLMs
21mType-Constrained Code Generation with Language Models
27mDecomposable Type Highlighting for Bidirectional Type and Cast Systems
49mImproving Accessibility through Leveraging Large Language Models (LLMs)
26mLanguage Model Agents Enable Semi-Formal Programming
30mEfficient programming with Scala and LLMs
49mA Practical Approach to Using LLMs: Peculiarities and Pitfalls
27mBuilding Useful Apps With Small, Local LLMs