Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
Talkat Data on Kubernetes Day North America 2023 · on Monday, 2023-11-06· at 09:15· W471ab, McCormick Place·
summary
Intuit reworks data pipelines onto Kubernetes to serve a self-serve Data Mesh that handles 100,000 data pipelines and 1000+ engineers; runtime uses Apache Spark and Flink on k8s with Argo, Kafka and S3, plus orchestration, checkpoint and DR layers. A domain-driven data map / data product model registers assets, SLAs and stewarded schemas (about 900 products in flight), and the platform visibly reduces discovery friction and speeds productized data delivery.
AI-written summary · about
chapters (6)
- 0:00Introduction to Intuit
- 2:19Data mesh concepts
- 5:32Data lake and data products
- 10:45Data map architecture
- 17:56Data processing platforms
- 23:02Q&A and closing remarks
related talks
47mData Mesh in Kubernetes
53mData Mesh in Practice - Assuring Data Quality at Scale
57mOur DataMesh Journey: data architectures for microservices
45mRethink Data Accessibility: Data Mesh meets Enterprise Integration
26mRe-architecting the Data Platform with Spark on Kubernetes
38mPlatform Building for Data Mesh - Show me how it is done!
43mUntangling Data Mesh: A paradigm shift in Big Data
22mNetflix Data Mesh: Composable Data Processing