What's happening in AI
summary
Company B10 runs inference for customers using shared model APIs and dedicated inference clusters, serving tens of thousands of GPUs daily and growing ~7–8x this year. A customer, Open Evidence, gets used by 55% of doctors and needs sub‑second answers, so B10 emphasizes multicloud failover to avoid cloud outages and to access scarce P200 capacity — an approach that maintains near‑zero downtime and cuts cost via scale. Talent and kernel‑level software remain the bottlenecks.
AI-written summary · about
chapters (13)
- 0:00Introduction to Baseten
- 0:56The importance of performance
- 3:26Inference strategies and choices
- 6:51The necessity of multicloud
- 9:36Challenges in AI infrastructure
- 12:00Outsourcing inference at scale
- 16:52Product evolution and flexibility
- 18:58Diversity of models and usage
- 21:09Open source runtimes
- 23:01The impact of agentic workflows
- 24:33Scaling a startup
- 27:31Open vs. closed source models
- 30:45Future prospects and bottlenecks
related talks
25mMaking AI Useful
20mAI Impact on the Backbone
25mAI-powered digital transformation
03mClosing Remarks
05mFrom One Node to Distributed Training and Inference. How the PyTo...
43mAI Knows the Board Better Than You Do
29mOpen Source Sovereign AI: Building Enterprise GenAI Platforms for Control & Comp...
18mSmall AI Teams with Huge Impact