Deploying Your Own Language Models: A Use Case Demo with Open Source...
summary
A question-answering system for Red Hat OpenShift on AWS (ROSA) is built from public docs, indexed with Qdrant vector embeddings and served as candidates to a generation model; baseline uses GPT-3.5, the open attempt fine-tunes Flan-T5 (783M) and deploys it with Ray Serve on a Kubernetes/OpenShift cluster. Demo shows nearest-neighbor retrieval, NDCG ranking, frequent link hallucinations (2 of 3 links wrong), and a human-feedback loop via Argilla to improve accuracy.
AI-written summary · about
chapters (18)
- 0:00Deploying language models
- 1:11Introduction
- 2:36LLMs and transformers overview
- 5:29Documentation search QA app
- 6:54Live QA demo
- 9:03Embeddings and vector search
- 13:18QA app architecture
- 14:43Vector databases for search
- 16:12Answer evaluation with prompts
- 17:56Adapting open LLMs
- 18:37Pre-training and fine-tuning
- 20:47Selecting open models
- 22:51Deploying Flan-T5 on Kubernetes
- 29:42Model performance metrics
- 30:47Human feedback importance
- 32:56Using feedback in apps
- 33:18Key steps to build app
- 37:24Q&A
related talks
40mDeploying Your Own Language Models: A Use Case Demo with Open Source Tools
18mOpen Hybrid Cloud & AI Developer Experience
31mDeploying Large Language Models with Ease
50mProduction-Ready GenAI with Open Models for Java Teams
43mDeploy Your Open-Source Language Model
01h 06mScaling personalization with AI
29mNavigating Innovation with Open Hybrid Cloud and Openshift AI
37mBuilding Generative AI Applications