Increasing Productivity and Reducing Costs in Big Data Ecosystems on AWS
summary
A Big Data project at BBVA scaled from processing 400 million to 15 billion transactions on AWS by adopting agile and DevOps practices. The talk describes the team-organization and technical changes used to automate and scale the pipeline—tools named include Java, Python, Jenkins, Rundeck, Puppet, Foreman, Luigi, CloudFormation, MongoDB, S3, EMR, Redshift, DynamoDB—and explains how uploads to S3 trigger fully automated jobs across 70 EMR clusters and 500 EC2 instances, performing 60K writes/sec to DynamoDB and exposing results via APIs.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 7:15Project challenges pilot
- 12:10Scaling infrastructure automation
- 20:49Process optimization orchestration
- 31:20Data lifecycle validation
- 38:07Results and ROI
- 44:53Q&A
related talks
29mCommBank's journey with AWS data marketplace
22mFireside Chat with AWS
42mWeb Development To Big Data. A Journey
52mPreparing data for reporting, ML, and AI on AWS
44mBest practices for analytics and generative AI on AWS(ANT302)
29mHow financial services companies derive greater value from data with AWS
31mAccelerating AI with Hyperscaler Platforms
22mHow Rocket Companies modernized their data strategy on AWS