Designing tests for ML libraries
summary
Shows concrete testing practices from HuggingFace: how diffusers and PFT run a stratified test schedule (fast per-PR, slow on main, nightly GPU runs), benchmark latency including torch.compile, and validate features like quantization, gradient checkpointing and parameter offloading. Describes using regression artifacts on the Hub, source-install tests against transformers, pytest fixtures plus pytest with caching to avoid rate limits, and measurable outcomes (10M monthly PPI downloads, 50k GitHub stars) proving these pipelines work at scale.
AI-written summary · about
chapters (14)
- 0:00ML library testing intro
- 2:59Why ML libraries need special testing
- 6:26Testing requirements
- 12:54Key challenges in ML testing
- 18:15Benchmarking model performance
- 21:28Code coverage for ML tests
- 24:17Regression testing outputs
- 26:27Handling known test failures
- 28:14Non-determinism and flakiness
- 31:44Caching strategies
- 35:20Floating-point comparison
- 37:31Code quality and workflow
- 39:20Choosing a testing framework
- 41:06Conclusion and key takeaways
related talks
56mTesting at scale
27mPyTesting the Limits of Machine Learning
32mLightning Talks
30mHow to Properly Test ML Models & Data
28mML-Driven Test Intelligence at Scale - What Works What Fails and Why It Matters
53mUnleashing the Black Box: Testing Machine Learning Systems
39mModern Data Pipelines Testing Techniques: A Visual Guide
30mUnit-tests for machine learning pipelines