Llama 3 at 1,000 tok/s on the SambaNova AI Platform
summary
Demo shows Llama‑3‑8B‑Instruct at 1,000 tok/s running on SambaNova’s SN40L RDU with a three‑tier memory stack; measured time‑to‑first‑token ~0.19s and end‑to‑end ~0.65s. Workshop builds a RAG Q&A using LangChain, Unstructured/pyPDF loaders, E5‑large‑v2 embeddings, ChromaDB vector store and a Streamlit/Jupyter app; a 15‑page PDF splits to 89 chunks, embeds in ~4s, and retrieval + LLM generation returns grounded answers instantly. Samba‑1 CoE (92 experts) and API keys are used.
AI-written summary · about
chapters (8)
- 0:00Introduction and setup
- 4:26Overview of Sova platform
- 8:27AI model evolution
- 12:00Composition of experts
- 18:34Hardware architecture
- 21:30Live demo: Samba 1 Turbo
- 25:17Hands-on coding exercise 1
- 43:46Hands-on coding exercise 2
related talks
29mUsing LangChain and LlamaIndex to customize LLMs...
48mBringing Llama 3 to Life
52mHW, SW, Performance and Costs for Llama-2 70b and Mixtral 8x7b LLM Inference with Low...
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
49mTame your Llama, run AI in Java
40mOpen Source LLMs in the Cloud: Scalable Solutions
27mAI Demystified Unveiling the Intelligence Behind the Machine
29mLeveraging open-source LLMs for production