Keynote: Pre-training and Fine-tuning of Code Generation Models
Talkat KubeCon + CloudNativeCon + Open Source Summit China 2023 · on Thursday, 2023-09-28· at 09:55· 3层 301明珠厅| 3F The Pearl Hall 301·
summary
A 15B code model called StarCoder trains on permissive‑license GitHub plus issues, commits and notebooks with deduplication, decontamination and PII removal and uses Megatron‑LM, multi‑query attention and fill‑in‑the‑middle; it reaches parity or better than Codex on Python and supports 80+ languages, fine tunes with PEFT/LoRA instead of full retraining, and deploys via Text Generation Inference and Kubernetes endpoints serving about 12,000 users after training with 15B parameters on 512 GPUs for 24 days with an 8000-token context.
AI-written summary · about
chapters (11)
- 0:00Code generation models intro
- 2:07Hugging Face’s open code models
- 3:11Open code model principles
- 4:36StarCoder training data
- 7:43StarCoder architecture and training
- 8:46Fine-tuning with PEFT
- 9:28BigCode ecosystem
- 9:49VS Code extensions
- 10:09Text Generation Inference
- 12:15Kubernetes for Hugging Face
- 13:42Model development behind scenes
related talks
36mUnleashing Generative AI: Improving Developer Productivity in SRE
43mKeynote: Applied NLP in the age of Generative AI
57mKeynote: Using LLMs and generative AI to fix software vulnerabilities
40mLLM for Coding, the State and lnitiative
25mTraining Large Language Models in the Open
01h 09mCodex presentation & demo
11mGen AI-Assisted Code Authoring
01h 22mBeyond Decoding: Meta-Generation Algorithms for Large Language Models