Shopify's Ray-Powered Approach to Multimodal AI in E-commerce
summary
Shopify builds a production pipeline that fine-tunes vision LLMs to map unstructured titles, descriptions and images into a 12,000-node product taxonomy with 40,000 attribute values, improving precision and recall across every level. Training and experiments run via SkyPilot on top of Ray, evaluation uses Ray actor pools for millions of batched LLM calls, and production serves low-latency NVIDIA Triton endpoints plus a high-throughput Dataflow+Kafka path — handling 50–70 million predictions/day with fine-tuned vision LLMs and SkyPilot + Ray actor pools showing measurable business impact.
AI-written summary · about
chapters (12)
- 0:00Introduction and speakers
- 1:32Why multimodal LLMs for Shopify
- 6:07Shopify product taxonomy explained
- 9:18Multimodal model training approach
- 15:57Fine-tuning process with Ray
- 17:44Evaluating model performance
- 20:12Real-time low-latency serving
- 22:18High-throughput batch serving
- 24:04Impact on Shopify’s ecosystem
- 25:08Q&A on model fine-tuning
- 27:17Vision model choices and trade-offs
- 29:04Managing taxonomy updates
related talks
37mLeveraging Generative AI for Enhanced E-commerce
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
16mKeynote: An Open-Source Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM
15mBuilding Reliable Browser-Use Agents for Storefront Optimization
22mRay, a Unified Distributed Framework for the Modern AI Stack
50mFive lessons Etsy learned building on Microsoft AI
16mGen AI For Retail Data Management
29mHow GenAI is transforming e-commerce catalogues