From CRM to DataLake with K8s and microservices

on YouTube ↗38m

" summary

Describes migrating a growing CRM into a Data Lake architecture using Kubernetes and microservices to handle integrations with multiple internal and external services. Covers building analytics and predictive models without overloading the production system by separating data ingestion/storage (Data Lake) and processing workloads on K8s-based microservices.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 3:14Monolith CRM problems
3. 7:10Moving reports to lake
4. 10:05Greenplum analytics choice
5. 13:14Kafka for real-time
6. 15:58Kubernetes orchestration
7. 17:51MinIO data storage
8. 19:17Final architecture
9. 23:20Q&A

" related talks

• 31mA Journey of Migrating Data Platform from Data Center to CloudLei Gao · Scale By the Bay 2021
• 43mData Lake beyond the BuzzwordMatheus Espanhol · QCon São Paulo 2018
• 51mHow to Build a Modern Data Lakehouse?Piotr Kalinowski · 4Developers 2022 Online
• 55mDesigning performant and scalable data lakesRukmani Gopalan · SQLBits 2020
• 32mModernize Your Data Integration Architecture For Better AnalyticsAdam Mayer · Big Data LDN 2022
• 34mTales of Data Architecture EvolutionJosef Goldstein · Build Stuff 2022
• 34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data MeshRakesh Subramanian Suresh, Jainik Vora · Data on Kubernetes Day North America 2023
• 42mSolve Big Data Challenges with Data LakeMehdi Hadji · BlaBlaConf 2021