From Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
summary
Company converts logged DQL queries (redacted) into a fine‑tuning dataset by reimputing plausible values and generating NL annotations with an LLM, clusters examples by complexity, source and semantics, and evaluates with executability, retrieval score, RO score and tree edit distance. Fine‑tuning small models (500M–8B) via LoRA on Hugging Face costs ~$30 and 24 hours on one GPU, yields ~70% → 90% executability versus GPT‑4 prompting, is two times faster, uses fewer tokens and is cheaper overall.
AI-written summary · about
chapters (11)
- 0:00Fine-tuning LLMs for query languages
- 1:07Problem definition and DQL challenges
- 3:58Prompting vs fine-tuning
- 6:48Prototyping pitfalls and VIP evaluation
- 8:15Evaluation metrics for query generation
- 10:21Synthetic training dataset
- 13:27Representative test set
- 15:57Fine-tuning setup and model selection
- 17:43Prompting vs fine-tuning results
- 19:09Key takeaways
- 19:52Q&A
related talks
29mLocal LLM fine-tuning: a practical example
31mFine-tuning LLMs: A Cost-Benefit Analysis for Businesses
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
25mFinetuning Large Language Models (LLMs) for Security Log Detections
30mLLMs: Beyond the Hype - A Practical Journey to Scale
09mTailoring Llama for Your Needs: Fine-Tuning & Evaluation
31mCustomizing LLMs: A Guide to local Fine-Tuning...
01h 30mUsing LLMs to improve your Search Engine