LLM for Coding, the State and lnitiative
summary
Claims adoption numbers: GitHub Copilot yields about 30% acceptance rate for completions while a China internal tool sees ~80% internal adoption; model sizes move from tens to hundreds of billions and bigger models slow down. Key engineering facts: heavy use of retrieval-augmented generation combining vector and keyword search then ranking to pick ~10 context records; synthetic-case generation and a self-improving StarCoder-15B data flywheel are practical. Performance limits include real-time needs under 200 ms latency and poor scores on realistic evals like s-bench even for GPT‑4.
AI-written summary · about
chapters (14)
- 0:00Introduction and background
- 1:49History of code generation
- 6:34Developer needs and AI tools
- 12:00Core elements of AI coding tasks
- 15:45Code vs natural language data
- 18:38Synthetic data for code training
- 21:50Intelligence flywheel concept
- 23:18Evaluation challenges in AI coding
- 26:32Code completion and speed
- 27:38Retrieval augmented generation
- 31:19Multi-agent systems in coding
- 34:54Agent techniques and SOP
- 35:36Open source data and models
- 38:23Personal use cases and evaluation
related talks
45mProgrammer’s Apprentice Season 2: Future Directions in AI-assisted Coding
40mKeynote
48mMastering the AI Shift: Adaptive Strategies for Software Evolution
10mWhat goes into the making of LLMs
16mThe LLM Leaderboard: Benchmarking AI Coding Models
33mPractical Evaluation of LLMs for Code
57mKeynote: Using LLMs and generative AI to fix software vulnerabilities
01h 05mThe Next Coding Horizon - Proactive LLMs for Enhanced Productivity