How I built an entire data platform by myself that thousands of data engineers use in one year
summary
Claim: one person builds a full data platform in a year that supports thousands of users, with stepwise architectures and clear tradeoffs. Concrete stack: Tabular, Starburst/Trino querying Iceberg, AWS Glue for Spark streaming, Confluent Kafka, Astronomer for Airflow, CloudFront paywall, Clerk auth, Stripe subscriptions. Metrics and surprises: 50→350 students, $100,000 single-day revenue, ~5–6k assignments, AI grading handling ~90% of homework. Major mistakes: $88k domain, rolling auth and streaming instead of using services.
AI-written summary · about
chapters (7)
- 0:00The one-person empire
- 4:12V1: The no-code architecture
- 7:47V2: The low-code architecture
- 10:42V3: The baby platform
- 14:04V4: Current architecture
- 19:27Lessons and mistakes
- 23:53Future of data engineering
related talks
35mBuilding a data analytics platform on strong opinions held loosely
21mData-driven approaches in a technology startup
50mMasterclass: Building a data stack in 2021
19mLessons from Building a Large-scale, Multi-cloud Data Platform at Databricks
46mThe challenges of data science for veteran media organizations
30mData Engineering building your BI infrastructure from scratch
21mHow Not to Build a Data Platform Lessons from Failure to Success
45mBuilding an Analytics Platform