Data Warehouse, Data Lakes, What's Next?
summary
Alibaba uses Flink to run real-time ML pipelines and builds Hologres as a unified storage layer to avoid islands of HBase/Cassandra/ClickHouse. Hologres ingests tens of millions of events/sec, exposes data immediately to serving and analytic queries via the Postgres protocol, and supports stream joins and online training (Alibaba reports a 20% sales uplift from an online model). Engine claims: order-of-magnitude faster than HBase for point lookups and 3x faster than Greenplum on analytics through disaggregated storage, async IO, cooperative scheduling and vectorized execution.
AI-written summary · about
chapters (9)
- 0:00Introduction and background
- 3:36Real-time ML pipeline overview
- 10:19Lambda architecture complexity
- 14:39Unified storage challenges
- 17:15Hybrid processing paradigm
- 20:05Hologres design goals
- 25:15Hologres architecture breakdown
- 34:12Benefits of Hologres
- 34:54Summary and conclusion
related talks
38mKeynote | Data Warehouse, Data Lakes, What's Next?
50mApache Hudi : The Data Lake Platform
31mBuilding large scale, transactional data lakes with Apache Hudi
25mBuild open format data lakehouse
40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data Lakes
34mCatalogs, Iceberg, and Delta: Building the Foundation for Lakehouse Analytics
31mPanel: Open Data Lakes
50mFast Analytics On Big Data