Scaling ONNX and TensorFlow model evaluation in search
summary
Vespa evaluates ONNX and TensorFlow models on content nodes so data stays local, not sent over the network, and a benchmark with a ~200k‑parameter neural reranker shows this matters: 1,000 results per query with ~500 floats each hits a 10 Gbit network ceiling around ~300 qps, while external model servers saturate even earlier (flat under 20 concurrent clients). Importing models into Vespa’s tensor language removes RPC overhead, increases total throughput, and lets adding content nodes or rewriting more documents scale quality (with Amdahl‑style diminishing returns).
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:27ML models in classification, transformation
- 2:37ML models in search, personalization, ads ++
- 6:55Vespa at verizon
- 8:18Vespa - core features
- 8:40Performance at scale
- 10:32TensorFlow, ONNX and XGBoost integration
- 12:12Converting computational graphs to Vespa
- 14:22Benchmark - recommendation system
- 15:29Scaling up model inference performance
- 18:00Increasing number of evaluated results
- 19:17To conclude
related talks
40mVespa.ai’s Personalized Search: Advanced Ranking & Tensor framework
22mTensor arithmetics in search and ranking for Ecommerce.
36mAccelerating Machine Learning at Scale with
26mAutomated Machine Learning Performance Evaluation
23mThe Race to the Bottom: Low Latency in the age of the Transformer
28mServing a Frontier Open Model with Ray & vLLM
37mAccelerate and Autoscale Deep Learning Inference on GPUs with KFServing
32mScaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve