Codex presentation & demo
summary
A large transformer model trained on public GitHub code generates code token-by-token and often produces syntactically valid completions; evaluation on the HumanEval suite (164 Python problems) shows pass@1 ~30% for the biggest model and pass@100 ~72%, while harder APPS competition problems sit below 1% at pass@1. The system runs in a playground with temperature sampling (demo shows diverse outputs), benefits from unit-test filtering, follows scaling laws for loss, and handles boilerplate and common libraries much better than novel logic.
AI-written summary · about
chapters (8)
- 0:00Start
- 3:00Recurrent Neural Networks
- 4:12Sequence To Sequence Learning
- 8:07Attention Modules
- 18:40Scaling Laws
- 53:21Sensitivity to Small Changes in Wording
- 1:05:17Is It Correct Only for Things That Are Part of the Training Data or Does It Extrapolate beyond the Training Data
- 1:06:27Remarks
related talks
15mKeynote: Pre-training and Fine-tuning of Code Generation Models
39mTraining AI To Code Using The Largest Code Dataset (Project CodeNet)
49mBen Hall & Barbara Fusinska
01h 02mOpenAI Codex Masterclass
51mOpenAI and the AI that may overtake humans
11mGen AI-Assisted Code Authoring
33mUsing Transformers.js to Bridge the Gap Between AI Research and
50mProduction-Ready GenAI with Open Models for Java Teams