Building a platform to process £34bn in card transactions
summary
Platform processes £34bn of card volume by ingesting heterogeneous files (2 KB to GB) from 20+ sources and 500k files per day, normalizing 1,000+ schemas into chunked Arrow outputs using Apache Arrow, running on GCP (GCS, Pub/Sub) with Kubernetes HPA + cluster-autoscaler (≈300 pods across 40 nodes at peak). PCI-sensitive PANs are decrypted in-memory, masked and routed under PCI DSS Level 1 controls. Outcome: self-serve schema registry, domain-owned pipelines, and autoscaling that meets strict SLAs.
AI-written summary · about
chapters (28)
- 0:00intro
- 0:27preamble
- 0:49about dojo
- 1:55taking a card payment is complicated and highly regulated
- 3:42challenges
- 6:39abstracting away the complexity
- 8:14other data domains as well
- 8:46data infrastructure generations
- 10:47challenges: central data platform
- 12:52what's next
- 15:11data mesh
- 17:14platform overview
- 19:31connectors overview
- 21:55pci dss level 1
- 23:04pci management process
- 23:48pci platform overview
- 25:45autoscaling challenges
- 27:46autoscaling triggers
- 28:40cluster autoscaler
- 30:48using hpa to elastically scale
- 32:00transformation platform overview
- 34:05schema registry
- 35:33state management
- 37:23real time file monitoring
- 37:58instrastructure observability
- 38:47analytics platform
- 39:31end-to-end file monitoring
- 40:33thank you for your time!
related talks
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions
10mBuilding a Data Platform at Assembly Payments
24mRetail Transactions at scale for AI - From PoC to production
29mBuilding a Batch Processing Platform...
19mAdoption of Big Data in Securities Processing Industry A Developer's Perspective
22mData Divide: End User’s Cloud Native Journey to Distributed Databases
30mData Platform Infrastructure at Rivian: A Deep Dive
22mBlack Friday Brilliance: Managing a Billion Transactions with Tech, and Teamwork