Ray Across the Stack Fine Tuning Qwen and Serving 3D Generation at Autodesk with AWS Trainium and In
summary
Autodesk fine-tunes Qwen3 on AWS Trainium, using Ray Train and Hugging Face Optimum Neuron to shard weights across 32 NeuronCores with TP32 sharding and runs 8B and 32B models; best runs hit ~90,000 tokens/sec and about 14¢ per 1M tokens by increasing sequence length, but compilation caches, dataset staging, communicator env vars and observability tooling break or add cost. Serving moves from GPU to Inferentia2 for static-shape workloads, showing time-to-first-token parity at ~80% cost and suitability for 4B diffusion 3D models.
AI-written summary · about
chapters (13)
- 0:00Introduction and overview
- 1:53Ray in Autodesk pipeline
- 4:05Switching to Trainium hardware
- 6:33Fine-tuning configuration choices
- 9:00Fine-tuning campaign results
- 13:38Trainium vs GPU comparison
- 16:25Cost and parity insights
- 18:30Trap and observability tips
- 19:54Custom ML accelerators iceberg
- 20:14Inference and kernel optimization
- 24:46Inferentia vs GPU performance
- 27:01When to use Inferentia 2
- 28:26Closing remarks
related talks
01h 45mRun Frontier AI at Home
31mFaster Model Serving with Ray and Anyscale
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
22mRay, a Unified Distributed Framework for the Modern AI Stack
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
22mHow Zoox Accelerated Autonomous Driving with Ray
32mAnyscale's Ray Data: Revolutionizing Batch Inference