Kimberly Leyenaar, Broadcom, and Steen Graham, Scalers AI | SC23
summary
Broadcom and Scalers AI describe running inference on 70B Llama models and fine-tuning Llama 2 on a PubMed corpus of tens of millions of records in a heterogeneous, distributed cluster (H100s, A100s, Dell PowerEdge variants, Broadcom Ethernet). Fine-tuning completes in days once the cluster is ready, but building the cluster (data ingest, munging, storage IOPS, networking) takes far longer. They demonstrate GPUDirect to bypass CPU bottlenecks, publish sample code, and outline scale lessons for a 500,000-node cluster.
AI-written summary · about
