Text-to-Struct: Fine-tuning SLMs for Query Intent
summary
Fine-tuning a 4B SLM (Gwen) via LoRA/QLoRA and knowledge distillation from a Sonnet 4.6 annotator turns raw user or agent prompts into Structured Semantic Query outputs; training uses ~6.5K generated+curated queries, then activation-aware quantization shrinks the model from ~8GB to ~2.5GB and doubles throughput (≈50→100 tokens/s). The stack uses vLLM for KV and prefix caching, produces query enrichment plus multi-vector query expansion, reduces costly 10–20s LLM tails, and enforces abstention when no evidence exists.
AI-written summary · about
chapters (8)
- 0:00Introduction to query intent
- 7:57Query intent challenges
- 11:09Initial prompt engineering solution
- 15:00Evaluating fine-tuning alternatives
- 16:47LoRA fine-tuning process
- 23:55Custom evaluation framework
- 29:53Results and performance metrics
- 33:29Q&A session
related talks
21mFrom Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
41mFrom Natural Language to Structured Solr Queries using LLMs
29mLocal LLM fine-tuning: a practical example
46mBeyond Basic Prompting: Supercharging Open Source LLMs with LMQL's Structured Generation
51mQuery Understanding With Large Language Models: Techniques and...
40mSurviving the Agentic Hype with Small Language Models
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
24mConversational Search with LLMs