~/talk

AI-Powered Impact-Based Testing for Faster Safer Releases

on YouTube 52m
Talkat TestMu Conference 2026 · 2026-08-19..08-21· [ai-assisted-development, testing]

speakers

name:Srikanta Sahoo
links:none on file
name:Navneet Goyal
linkedin:navneet-goyal-91031210

resources

video:https://youtu.be/XaUFnygny9A
slides:none on file

summary

Smart Engine analyzes each commit using 20+ signals — structural imports and call graphs, semantic embeddings (BGE‑M3), historical failure patterns, trust scores and dynamic risk — then ranks tests with SGD + LightGBM and returns a sub‑2s impact set to CI. Results show ~90–95% test reduction with recall maintained and accuracy rising from ~85% week one to ~98% at six months on a 100,000+ test codebase. FastAPI + Python backend pgvector storage and explainable scores

AI-written summary · about

chapters (55)

  1. 0:00Start
  2. 0:06Session Opens - AI-Powered Impact-Based Testing
  3. 1:07Running the Whole Suite on Every Change Is Expensive and Slow
  4. 2:08Speaker Intros - Srikanta Sahoo and Navneet Goyal, Principal Software Engineers at UKG
  5. 2:39The Opening Question: What If Every Code Change Told You What to Run?
  6. 3:10Ten Lines Changed, Thousands of Tests Queued
  7. 5:23A Decade of Change: Smaller Commits, More Frequent Deployments
  8. 5:53What Didn't Change: How We Run the Regression Suite
  9. 7:24Test Execution Becoming the Delivery Bottleneck
  10. 8:56The Right Question: Do All 50,000 Tests Have Equal Probability of Impact?
  11. 10:01The Real Cost of Full Regression Is More Than Compute
  12. 10:32The Cost Teams Don't Count: Developer Context Switching
  13. 12:07The Regression Paradox: Two Bad Choices
  14. 13:10Wanting Confidence That Can Be Explained and Defended
  15. 13:40Test Selection Isn't New - Tags, Priorities and Severities
  16. 14:42Why Each Breaks at Scale: Tags Need Discipline, Mappings Go Stale
  17. 15:13The Missing Ingredient in All of Them: Understanding the Change
  18. 16:46The Shift From Regression to Intelligent Regression
  19. 17:17Identify Impact, Rank by Risk, Execute the Subset
  20. 18:20The Types of Impact: Direct, Indirect, Semantic, Historical
  21. 19:21Critical Path: A Small Change to Payments or Auth Deserves More
  22. 20:55Combining 20+ Signals to Make the Selection Change-Aware
  23. 21:27AI Recommends; Engineering Policy and Risk Controls Still Govern
  24. 21:58Six Signal Categories: Semantic, Structural, Historical, Quality, Dynamic, ML
  25. 22:30Structural: Import Analysis, Call Graph, Package Coupling
  26. 24:03Semantic Signal: Comparing Business Meaning, Not Just Location
  27. 24:34The Bank Analogy - River Bank vs the Bank With Your Money
  28. 25:36Historical Signal: What Code and Tests Change Together
  29. 26:06Failure History and Separating Real Defects From Flakiness
  30. 27:10Quality Signal: The Trust Score and When to Widen Selection
  31. 27:43Explainability: Why Each Test Was Selected
  32. 28:13Dynamic Signal: Classifying the Type of Change
  33. 28:44Risk Score: Estimating the Blast Radius
  34. 29:49The Two ML Models: SGD for Linear, LightGBM for Non-Linear
  35. 30:50A Real Example: Changing Password Length Validation
  36. 31:53One Test Scored: Semantic 0.92, Dependency 1.0, Historical 0.85
  37. 32:55The Team-Defined Threshold Selecting Three of Four
  38. 33:55It Augments Your CI/CD - It Doesn't Replace It
  39. 34:25The Stack: FastAPI, Python, BGE-M3 Embeddings, LightGBM, pgvector
  40. 35:27Continuous Learning From Every Change and Every Execution
  41. 36:58Accuracy Over Time: ~85% in Week One, ~98% at Six Months
  42. 38:02The Measured Benefits: 90-95% Test Reduction With Recall Maintained
  43. 38:33Validated Against a 100,000+ Test Codebase
  44. 39:36The Headline Isn't Fewer Tests - It's Faster Confidence
  45. 40:07The Production Question: What If the AI Misses a Test?
  46. 40:38The Principle: The Model Proposes, Policy Governs
  47. 41:09Lessons From Early Evaluation: The Skipped-Build Problem
  48. 42:42Auto-Detecting Newly Written Tests So They Always Run Once
  49. 44:16The One Idea: Running the Right Tests With Explainable Reasoning
  50. 44:46Q&A: Should We Still Run Every Test for Every Release?
  51. 45:50The Recommendation: Every Nth Run Should Be a Full Run
  52. 46:51Q&A: Should Every Iteration Be Deployed?
  53. 48:25Q&A: Generating Enterprise Test Data
  54. 48:57Q&A: When an Agent Bypasses an Important Business Control
  55. 50:29Human Diligence Still Needed in the First Quarter

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.