Building a data analytics platform on strong opinions held loosely
summary
A three‑to‑four‑person team builds and operates a self-service data analytics platform using Terraform, Snowflake, dbt and Airflow, serving ~620 active users and 1,000+ indirect consumers. The platform enforces strong opinions held loosely: data mesh ownership that grows to 58 data domains, 230+ databases and ~3,000 pull requests, everything-as-code modules for provisioning, and multi‑env release brakes. It scales, supports billion‑dollar business decisions, and survives large Snowflake upgrades (5,000 resources, ~4 hours downtime).
AI-written summary · about
chapters (10)
- 0:00Introduction
- 5:40Strong opinions loosely held
- 9:15Data mesh for scalability
- 14:14Three years of data mesh
- 16:02Everything as code
- 22:03Environments speed and control
- 27:45Diversity and platform maturity
- 28:48Minor opinions success
- 31:40Key takeaways
- 34:05Q&A
related talks
17mDemocratising data analytics
39mBREAKING YOUR DATA TEAM OUT OF THE SERVICE TRAP
29mHow I built an entire data platform by myself that thousands of data engineers use in one year
28mUsing Dataops To Build And Manage Data Products On Snowflake
10mBuilding a Data Platform at Assembly Payments
19mKickstarting a greenfield data project
27mRecipe for Building a Discoverable and Governed Data Platform.
48mBuilding a data mesh