Lightning Talk: From Hugging Face To Handheld: Scaling LLM Deployment W...
summary
Converts a custom Hugging Face checkpoint into a cross‑platform LiteRT IR in under a minute, runs quantization in ~40 seconds, and pushes the packaged model to a phone to run on-device GPU successfully; workflow uses the LiteRT Torch toolchain, Edge Quantizer, and LiteRT runtime with PyTorch FX lowering and an MLIR backend, supporting CPU/GPU/NPUs and producing a single TRM bundle (model, tokenizer, metadata). Demonstrates enabling QWEN0.6 in 20 minutes and end‑to‑end verification via terminal or Colab.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:25Exporting and quantizing
- 2:48Running on phone GPU
- 4:51Lite RT rebranding
- 6:33Model export pipeline
- 9:21NPU compilation benefits
- 10:45Conversion tools and APIs
- 11:29Edge Gallery app
- 12:09Resources and closing
related talks
01h 40mFrom model weights to API endpoint with TensorRT LLM
31mHugging Face ecosystem for Local AI/ ML
16mLightning Talk: Efficient Inference at the Edge: Performance You Need at the Lowest...
11mLightning Talk: Accelerating Experimentation...
18mLightning Talk: TorchRL - RLHF Support
23mHugging Face Accelerate: Making Device-Agnostic ML Training and Inference Easy...
13mLiteRT.js, Google’s high performance WebAI runtime
10mLightning Talk: Debugging Edge AI on Zephyr and Lessons Learned