Happiness U-Curve: Navigating the AI Validation Bottleneck with Conformance Testing and(…)
summary
Argues that validation — not generation — is the AI bottleneck and demonstrates a concrete dual-track approach: property-skeletons plus automated proof attempts and property-based tests. On an AWS authorization system, semantic input generation cuts test-engine dev time by 75%, raises coverage 4% and finds 28% more production issues. On Apache Spark and TVM, agentic instantiation of about 5,000 PBTs across 212 operators yields fuzzing-level coverage, more diverse operator exercise, and shift from invalid-input noise to actionable bugs like numerical instability and API mismatches using a well-formed input generator and dual-track alignment.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:49Keynote begins
- 2:33AI validation bottleneck
- 9:22Validation bottleneck in AI
- 10:47Keynote agenda
- 11:32Fuzzing vs property testing
- 13:16Testing data systems at scale
- 17:33Semantic input generators at AWS
- 20:04Solving specification problem
- 21:51Dual-track validation approach
- 22:54Property patterns in verification
- 24:19Property skeletons for proof engineering
- 27:54Proving properties in Lean
- 31:51PBT with skeletons
- 36:49Testing DL compilers with PBT
- 39:03Future of AI validation
- 46:31Happiness U-curve title
- 47:37Acknowledgments
- 49:02Q&A
related talks
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
41mAI-Augmented Testing: How Generative AI, Prompt Eng Turn Testers Into Superheroes
23mBeyond Tests What to Verify in AI Generated Code
37mAgentic AI and the Next Decade of Quality Engineering
31mAgentic AI Changed What We Test, And How We Test It
01h 05mPanel Discussion: Building Confidence in AI-Driven Testing Decisions
23mBeyond Tests: What to Verify in AI-Generated Code
39mAI-assisted development: Common pitfalls and smarter practices