SLM-powered retrieval to scale freelancers matching at Malt
summary
Malt builds a neural retriever using SLMs as a frozen 113M backbone with a 7M trainable head, stores embeddings in Qdrant, and serves approximate nearest-neighbor queries from a vector database. Quantization and graph optimizations shrink model size ~4x and bring CPU inference to ~17 ms; end-to-end latency drops from tens of seconds to a 3 s max. A/B testing shows improved conversion and the retriever removes prior suboptimal candidates, so the system works in production.
AI-written summary · about
chapters (22)
- 0:00Malt’s freelancer matching intro
- 1:48Existing recommender limits
- 3:13Project briefs and profile data
- 4:59BM25 retrieval shortcomings
- 6:03Neural retrieval and latent space
- 7:49Right multilingual language model
- 9:58Training models for skill matching
- 11:25Custom document architecture
- 13:35Token embeddings for sections
- 14:59Section encoding and trainable head
- 16:25Pooling embeddings to vectors
- 17:07Contrastive loss with historical data
- 19:36Data quality and bias issues
- 21:02Triplet loss and negative mining
- 22:47Latent space offline evaluation
- 24:10Embeddings in vector database
- 25:17Choosing vector database
- 26:44CPU inference optimization
- 28:50Quantization and graph results
- 29:55Production latency improvements
- 30:58A/B test conversion gains
- 31:19Future hybrid retrieval work
related talks
26mEfficiently serving LLMs at scale.
12mPreview: ML Serving
01h 06mScaling personalization with AI
20mSelf-hosting AI LLMs - a beginners guide
14mScalable Solutions for Running Large Language Models
28mNavigating LLM deployment: Tips, tricks, and techniques
24mLLM Inference Optimization From Simple To Sophisticated
36mAccelerating High-Performance Machine Learning at Scale i...