Retail Transactions at scale for AI - From PoC to production
summary
A production pipeline ingests roughly 1 billion transaction files per year from 2,000+ stores across 70+ countries, converts ugly XML with 340+ fields into typed tables using Spark, Scala and a Cascade validation approach, tracks field-level error rates and missing records, and delivers forecasting, recommendations and provisioning models; building the pipeline and governance takes ~2 years while a forecasting model ships in under two months, and the system largely works once monitoring and referential enrichment are in place.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:08Data engineering vs AI
- 3:39AI use cases in retail
- 5:26Project context and challenges
- 8:16Data enrichment issues
- 9:39Data governance role
- 12:54Data quality and monitoring
- 14:00Pipeline design and errors
- 16:57Data lake architecture
- 18:27Cascade validation
- 20:13AI and data engineering
- 21:37Takeaways
related talks
13mScaling AI Systems for High-Volume Transaction Processing
39mProductionising Data Science at Scale
51mReal-time Fraud Detection: Challenges and Solutions
22mData Ecosystem to power AI
15mScaling AI-Driven Financial Services: Ethical ML at Scale
28mBeyond the Balance Sheet: Transactional Data as a Window to Company Health
08mReal-Time Data Systems: Speed, Scale, and Resilience
59mHow Siemens & Petco drive enterprise transformation with data & AI