Scaling to billions of requests the serverless way at Capital One
summary
Capital One replaces heavy Spark clusters with a serverless pipeline built on AWS Lambda and Amazon Kinesis, using Amazon API Gateway for APIs; reusable source/processor/sink lambdas plus an SDK add deduplication, secret management, metrics and retries. Kinesis shard throughput (~2,000 tps per 1KB) and the iteratorAge metric drive autoscaling, multi‑region failover runs active/passive without always‑on servers, and the end result claims roughly 1/10 the cost of the prior Spark stack while handling thousands of TPS and supporting real‑time insights for millions of customers.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:15Overview
- 0:42How many of you check your statements
- 2:29Big data engineering tools
- 7:28Server streaming
- 9:09Lambda
- 10:36Message Bus
- 12:05Reusable Functions
- 13:15SDK
- 16:14Scaling
- 17:19Logging
- 18:25Spark vs Streaming
related talks
25mDeep Dive on Real World Use Case in Capitalone on Serverless and How It Enables Its Partner's...
38mServerless Data Streaming on AWS
31mAccelerate business innovation with serverless applications
29mBuild highly scalable serverless applications on AWS
36mTransforming SaaS with Serverless Architectures
47mAdvanced Serverless Architectural Patterns on AWS
41mIs Serverless the End of Kubernetes?
49mServerless Architectural Patterns and Best Practices