Workshop: Choosing the Best Open Source LLM for Your Application
summary
Claims that leaderboards mislead and human supervision is essential for real-world LLM selection; benchmarking shows about one in six benchmark queries hallucinate. Demonstrates a live workflow using Label Studio plus a label-studio-ml inference server to plug multiple 7B open models into an annotation UI, extract structured JSON (invoices, dates, amounts), and feed webhooked analytics into a dashboard. The pipeline produces on‑the‑fly model predictions, human-corrected ground truth for comparison, and datasets ready for fine‑tuning; demo has occasional server hiccups but otherwise works.
AI-written summary · about
chapters (6)
- 0:00LLM evaluation challenges
- 9:41Benchmarks vs real-world use
- 13:54Human-centric evaluation workflows
- 22:28Connecting to Label Studio
- 31:24Live model comparison dashboards
- 45:25Summary and production
related talks
28mA community based open source LLM and what it can do for your business
42mOpen Source LLMs — Powering AI Accessibility and Innovation
15mOpenAssistant's Opensource LLMs Saga
25mSelecting the Best LLMs with Model Cockpit
28mNavigating LLM deployment: Tips, tricks, and techniques
33mOpening Keynote: LLMs, between fantasy and reality - the poin...
38mLLMs for Self-serve Analytics - Lessons Learned
40mInstructLab: Applying Open Source Methods to Building and Trai...