LLM Inference on RISC-V Embedded CPUs
Talkat RISC-V Summit North America 2024 · on Tuesday, 2024-10-22· at 14:35· Grand Ballroom G (Level 1)·
summary
Optimizes LLaMA.cpp for the RISC-V P extension and runs TinyLLaMA 1.1B on an Andes Voyager board with a quad-core X45MP, using 8‑bit and 4‑bit quantization plus SIMD (SMQA) microkernels; baseline token rate is ~1 token/sec using scalar code and rises to >4 tokens per second after 4–5× optimizations, while TinyLLaMA‑7B drops to ~1 token/sec; gemm/matrix‑modification stays the dominant cost and dynamic quantization still requires the floating‑point unit.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 3:05Development board setup
- 4:00RISC-V P extension benefits
- 4:56Optimization strategies
- 10:17Performance benchmarks
- 11:14Operations and perplexity
- 13:36Conclusions and outlook
- 14:17Q&A session
related talks
13mDemo: RISC-V as an Enabler of Heterogeneous Compute
29mLeveraging open-source LLMs for production
23mRunning an Open Source LLM on Open Source Hardware
17mMaking LLVM RISC-V Compiler More Performant for Everyone
51mPractical LLM Inference in Modern Java
22mAdvancing AI Computing With Optimized Matrix Multiplication Techniques With RISC-V CPU
40mHands-on TinyML Deployment on RISC-V: Building an Optimized Edge AI Pipeline - Corrected Audio
47mPractical LLM Inference in Modern Java