Role of Private Data Lake for Sustainable Business Growth
summary
Walton runs a private data lake on Kubernetes processing data from 7,700 outlets, ~10M customers and ~20M products, moving heavy analytics off OLTP to cut costs and scale. Pipeline uses Airflow + Kubernetes executor to schedule extracts, dumps raw files to S3, exposes tables via Iceberg on S3, and runs distributed ETL and ML with PySpark + Trino + ClickHouse; Kafka handles streaming. An inactive-number model (published on Hugging Face) plus fraud detection actually saves crores of taka and supports customer targeting down to upazila level.
AI-written summary · about
chapters (15)
- 0:00Walton's data journey
- 1:31Private data lakes for growth
- 5:27Data infrastructure on Kubernetes
- 7:57Data lake architecture overview
- 9:00OLTP vs OLAP explained
- 14:28Cost benefits of separation
- 16:35Key analytical dashboards
- 19:13Consumer profiling insights
- 22:54Machine learning for strategy
- 24:44Fraud detection models
- 29:47Kubernetes data lake architecture
- 33:00Airflow for pipelines
- 35:44Spark for analytics
- 38:39Iceberg table format
- 40:13Q&A
related talks
17mDemocratising data analytics
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake
43mManaging a Data Lake on AWS
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
26mBuilding a data lake at a bank #bbuzz
34mData Lakehouse: Who, Why and How
28mBuilding A Governed Data Lake
41mTransactional data lakes for your analytical workloads