Data at scale: How to deal with small challenges when they become massive
summary
Lazada (Alibaba Group) builds a centralized data platform that ingests everything in real time, enforces strict naming and a common data model, and exposes both batch and real-time versions of datasets; the platform hosts millions of tables, uses a proprietary PII detector to lock sensitive columns, applies column-level access controls, and auto-shuts pipelines for unused tables to save cost. Goals are usability, absolute stability (platform cannot break), efficient compute/storage, metadata/lineage search, an integrated IDE, and automated data-quality alerts.
AI-written summary · about
chapters (10)
- 0:00Start
- 1:29What Is the Big Deal with Big Data
- 3:26Alibaba
- 6:10Example of a Task
- 13:00Diminishing Rate of Return
- 21:00Goals of Our New Data Platform
- 24:08Data Warehouse
- 26:16Access Controls on Column Level
- 28:00Computing Queue and Storage Management
- 32:16Structure Your Common Data Model
related talks
48mFrom (Big) Data Mess to Data as an Innovation Enabler
41mRole of Private Data Lake for Sustainable Business Growth
20mData Technology Enabled AI Solutions
53mData Mesh in Practice - Assuring Data Quality at Scale
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake
37mThe perils of building a democratic data platform
29mData foundation for analytics and AI
32mPromoting a data driven culture in a world of microservices