Demo: Running Transformers on Semidynamic's "All-In-One" Vector and Tensor Unit

on YouTube ↗11m

" summary

Semidynamics runs Llama‑2 7B FP16 via ONNX Runtime on an R5‑controlled chip that integrates core, Vector and Tensor units with no external DMA. The Tensor unit alone yields a 170x speedup over scalar R5, and adding the Vector unit boosts overall acceleration to ~470x and balances workload roughly 55/45 vector/tensor. A dedicated "gazillion unit" streams memory to avoid starvation. The design is modular so performance scales by replicating accelerator tiles.

AI-written summary · about

" chapters (7)

1. 0:00Introduction and market trends
2. 1:03Challenges in AI hardware
3. 3:11All-in-one AI solution overview
4. 4:36Gazillion unit and data movement
5. 5:42Software stack and ONNX runtime
6. 7:08Demo: Running LLaMA 2
7. 9:52Future-proofing and scaling

" related talks

• 20mBoosting AI on Semidynamics RISC-V Cores with Custom Tensor Instr...Roger Espasa, José María Arnau · RISC-V Summit North America 2023
• 11mBoosting AI on Semidynamics RISC-V Cores with Custom Tensor InstructionsP Marcuello · RISC-V Summit Europe 2024
• 12mDemo: Boosting AI on Semidynamics RISC-V Cores with Custom Tensor InstructionsJosé María Arnau · RISC-V Summit North America 2023
• 10mBoosting AI on Semidynamics RISC-V Cores with Custom Tensor InstructionsRoger Espasa · RISC-V at embedded world 2024
• 28mAn Efficient Implementation of TensorFlow Lite for RISC-V VectorsMostafa Hagog · RISC-V Forum: Vector & Machine Learning 2021
• 23mRunning an Open Source LLM on Open Source HardwareEthan Sifferman · Latch-Up 2026
• 23mThe Race to the Bottom: Low Latency in the age of the TransformerMax Irwin · Berlin Buzzwords 2022
• 24mEnhancing Transformers: Accelerating Nonlinear Function Computation On RISC-V Vector ProcessorHeng-Kuan Lee, Simon Wang · RISC-V Summit North America 2023