Cloud Agnostic Data Platform with DBT, Trino, Iceberg and MinIO
summary
A cloud-agnostic pipeline that actually runs: Trino as a distributed SQL engine (with Trino coordinator + workers), DBT managing CI/CD, tests, snapshots and materializations (DBT-driven CI/CD with tests and snapshots), Iceberg tables stored on MinIO with Hive metastore (Iceberg tables on MinIO object store). Security enforces RBAC/PBAC across Trino and MinIO so access follows bronze→silver→gold roles, monitoring via JMX/Prometheus/Grafana shows queries and resource usage, and all components are open source and runnable locally or on Kubernetes.
AI-written summary · about
chapters (6)
- 0:00Introduction and architecture
- 3:06MinIO object storage
- 8:35Trino query engine
- 16:08Security and monitoring
- 19:43DBT for data operations
- 32:10Local deployment and conclusion
related talks
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
50mDesigning a scalable data platform based on Apache Spark/Iceberg
25mHow to build a platform for AI and analytics based on Apache Iceberg
31mBuilding a Data Infrastructure for AI/ML
34mApache Iceberg.
01h 12mHands-on with Apache Iceberg
29mLakeside AI: Bringing Models to Data with Open Lakehouses & Agents)
45mEnabling Data Sovereignty in AI Workflows through Data Mesh Architecture