Specs Don't Hallucinate: Deterministic Code in the Age of AI
summary
An experiment builds a Go task service using oapi-codegen and sqlc versus pure AI generation across greenfield and brownfield scenarios. Ten greenfield runs show AI plus cogen hits 5/5 success, finishes >2× faster and uses ~40% fewer input tokens. Sixty brownfield runs find mixed results: most features 5/5 for both approaches, but optimistic locking favors AI plus cogen and reduces AI-written review surface. Allowing AI to update specs works reliably and ready-made spec changes do not improve success.
AI-written summary · about
chapters (10)
- 0:00Introduction and disclaimer
- 3:04Deterministic code generation tools
- 7:49Research questions and experiment setup
- 10:01Demo project and scenarios
- 15:56Evaluation metrics and process
- 16:41Greenfield scenario results
- 17:31Brownfield scenario results
- 20:11AI vs human specification updates
- 22:47Conclusion and limitations
- 25:40Q&A session
related talks
37mSpec Driven Development: AI Assisted Coding With Doc at the Core
53mNavigating AI Native Development
26mCoding with AI
41mAI or Not AI? We Don't (Yet) Believe the AI That Replaces Devs!
39mAI-assisted development: Common pitfalls and smarter practices
39mBuild without limits: lessons learned from using generative AI dev tool
31mAI-Driven Development in Go
44mSpec-driven Development: How AI Changed Everything (And Nothing)