Inferencing at the edge and Fragmentation Challenges
summary
Argues that edge inferencing suffers from massive hardware and software fragmentation and quantifies the space by contrasting device classes from microcontrollers to edge servers, runtime approaches (Qualcomm neural processing SDK, DLC, TF light, Arm NN) and graph compilers (TVM, n graph, XLA, Glow). Identifies four levels of IR (graph, operator, shader/AST, assembly), recommends focusing on TensorFlow and Onix for broad coverage, and suggests reusing runtimes or common CPU backends to reduce duplicate effort and scale deployment.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:03Inferencing at the Edge
- 2:38Deployment Scenarios
- 4:02Application Development Options
- 5:58App Examples
- 8:04ARM NN Runtime Example
- 9:59Graph Compilers
- 11:49Framework Layer Cake - Example nGraph
- 14:46Projects Mapped to Four Levels of IR
- 16:31Frameworks, Platforms, HW, All Growing
- 17:31HW Diversity Is a Scalability Issue
- 18:12Runtime Components Map to Op Level IR
- 19:12Using Full Runtime as a HW Backend
- 20:40ARM Inferencing Edge Server
- 21:31Addressing Graph Compiler IR Fragmentation
- 23:15Addressing CPU Runtime Fragmentation
related talks
37mDeep Learning Neural Network Acceleration at the Edge
37mDeep Learning Neural Network Acceleration at the Edge
48mThe Power of Edge AI: Revolutionizing Future of Real-Time, Autonomous Processing
16mEdge AI needs to be for everyone: breaking barriers to adoption
21mUnlocking the potential of low-power embedded devices with Edge AI software
21mA New Architecture for the Intelligent Edge
06mAI Inference at the Edge: Beyond the Hype
36mApplied AI on the edge