Train, Evaluate, Repeat: Building a Credit Card Fraud Detection System
summary
Stripe faces three concrete problems and gives concrete fixes: pick global labels like chargebacks and card‑network flags such as TC40 rather than merchant heuristics; handle extreme class imbalance (actual fraud is <1% fraud) by training on all positives and downsampling negatives to a target positive rate (they pick ≈10% and see better PR/AUC and long‑term recall); and use counterfactual logging by uniformly allowing a fixed percent of blocked scores (uniform sample with 1/p weights) which yields low‑variance production estimates and prevents forgetting easy fraud while saving compute.
AI-written summary · about
chapters (14)
- 0:00Stripe and fraud challenges
- 2:26Defining fraud impact
- 4:52Fraud challenges to ML
- 5:12Effective fraud labels
- 9:49Balancing labels and business
- 10:10Class imbalance in fraud
- 13:22Training on balanced vs imbalanced
- 18:00Optimal fraud rate
- 20:03Unbiased production evaluation
- 21:50Counterfactual evaluation
- 24:40First unbiased attempt
- 29:39Uniform sampling attempt
- 31:48Key takeaways and optimization
- 33:13Q&A on fraud and bias
related talks
47mTrain, Evaluate, Repeat: Building a Credit Card Fraud Detection System
13mHarnessing Machine Learning to Combat E-commerce Fraud
40mFraud detection without labeled data
26mAI-Powered Fraud Detection in Cloud-Native Digital Payments
33mChris Stucchio of Simpl at Machine Learning Developers Summit 2019
25mScaling AI for high-stakes, real-time payments
45mAnti-Fraud in Reverse and the Use of ML Methods in It
37mThe Role of Machine Learning in Today’s Payment Ecosystems