Bird's Eye View of Open-Source AI Infrastructure
summary
Overview gives concrete tradeoffs and outcomes: Llama 2 70B = ~130GB weights so hosting needs big GPUs or heavy quantization, computed 100 billion embeddings warns API costs, and a small hackathon build did ~10 million embeddings. Open-source stack examples include uSearch (HNSW engine) and uForm (multimodal encoder) and claim real results: 1.3 ms CPU inference and 30–100k search inferences/sec depending on kernel bypass and indexing choices.
AI-written summary · about
chapters (20)
- 0:00Open-source AI infrastructure
- 1:33LLM frameworks like LangChain
- 4:47Proprietary LLM API costs
- 6:53Open-source LLM models
- 9:02Evaluating LLM models
- 12:55Deploying large language models
- 15:03Embeddings for search
- 18:32Vector search engines
- 22:29Modularity in AI projects
- 23:10Unum projects
- 27:44Networking for AI workloads
- 30:15StringZilla data processing
- 31:42Hardware for vector ops
- 33:06Contribute to open-source
- 33:49Q&A: Scalability vs. functionality in vector search
- 35:16Q&A: Accuracy vs. scaling
- 36:41Q&A: E5 embeddings performance
- 38:26Q&A: FPGA for AI workloads
- 39:55Q&A: CPU inference for uform
- 40:47Q&A: Embedding scaling laws
related talks
47mBoosting AI With OpenSearch: Vector Stores, Similarity Search, and Retrieval-Augme...
14mKeynote: Building AI with Open Source & Hugging Face
35mOpen-source AI: why it matters and how to get started
27mKeynote: Open Source in the Age of AI
42m3 Pillars of Open Source AI: Training, Inference, Agents
40mKeynote: Open-source Multimodal AI in the Wild
13mKeynote: To Nemotron and Beyond: Building the Open Source AI Ecosystem- 14mKeynote: Building AI with Open Source & Hugging Face