Beyond Basic Prompting: Supercharging Open Source LLMs with LMQL's Structured Generation
summary
LMQL enables interweaving prompts and Python for LMQL‑style structured generation, using logit biasing, type constraints and control flow to produce validated JSON and simple classifiers. Demo builds a Wikipedia RAG with a 1.7B 4‑bit open‑source model (llama.cpp) plus tests on a 135M model, showing heavy hallucination: factual correctness ~22%, faithfulness 25% and context recall ~5%. GPT‑4 runs two to three times better; LMQL is unmaintained but the logit‑processing ideas survive in Guidance and Outlines.
AI-written summary · about
chapters (11)
- 0:00LMQL and structured generation
- 4:44Structured data generation
- 9:22Simple RAG with LMQL
- 13:13Generation flow in Python
- 18:54Evaluating RAG with RAGUS
- 21:45Decoder strategies for better output
- 27:05RAG improvements with Wikipedia
- 32:23Small LLM performance limits
- 33:50Live demo and interaction
- 37:47Production with LMQL alternatives
- 39:51Q&A
related talks
30mPrompting Is Programming: A Query Language for Large Language Models
01h 49mAI engineering with open access LLMs that lie, curse, and steal
27mProductionizing Open Source LLMs
01h 02mA hacker's guide to open source LLMs
28mA community based open source LLM and what it can do for your business
55mLessons Learned From Implementing Generative AI Solutions
26mHow To Win Friends & Influence LLMs (with Prompt Engineering)
41mEnhancing LLMs with Graph Technology