Chris Harbert: Testing AI at the Scale of Big Data
summary
Responsible for running 5,000 models on streaming pipelines processing 2 million records per second, the team builds automated evaluation that computes precision/recall/fscore per model in CI, learns Pig/Hive/Hadoop to reproduce production queries, and routes a sampled percentage of live traffic into QA (A/B streaming to QA) to test real inputs. A data type change from M/F char to boolean silently skews predictions and is caught by the pipeline tests. Lessons: expensive full QA DB ($8M), reengineer models for scale, and test metrics not just outputs.
AI-written summary · about
chapters (9)
- 0:00Testing AI and Big Data
- 3:30AI evolution and applications
- 10:17Challenges in AI testing
- 11:21Big Data testing hurdles
- 13:48Testing AI at scale
- 18:02A/B testing Big Data models
- 21:15Real-world AI testing lessons
- 29:49Applying lessons to future tech
- 31:59Q&A
related talks
35mBuilding Data-Intensive Teams
59mContinuous Intelligence: Keeping your AI Application in Production
50mData-Driven Engineering
26mTesting a Data Science Model
28mAccelerating AI at Scale with Data
16mOperationalizing AI in Finance: Real-Time Analytics
01h 40mBuilding machine learning pipelines that scale
28mDataEngBytes 2023 - PER-T1-05 - Ellie Gholami