Evolution of a data platform for timely communications with 100+ million customers
summary
Describes building a data platform to support timely communications with over 100 million customers, explaining the business goal that requires data and the initial engineering requirements for the data platform. Covers the team's objectives at the start of the platform project and the approach to solving data-driven messaging at large scale.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:08Business challenge
- 2:38Old system problems
- 4:25Hadoop platform switch
- 5:31Data migration issues
- 7:58Development process optimization
- 11:04Calculation reliability improvement
- 14:45Data quality control
- 17:16Data mart updates acceleration
- 19:03Real-time processing implementation
- 21:50Migration results
- 23:59Q&A
related talks
44mFrom a hack to the data mesh, the evolution of data engineering...
51mHow to Build a Data Platform from Scratch
28mData Platforms Transformation - Why and How
23mFrom Start-Up to Scale-Up: Building a Data Team at a High-Growth Global Ed-Tech Company
31mA Journey of Migrating Data Platform from Data Center to Cloud
33mAccidental DataOps: How we matured our organization with data, and how you can too!
34mTales of Data Architecture Evolution
21mHow Not to Build a Data Platform Lessons from Failure to Success