The Bucket Brigade: Why S3 is the Missing Piece in your AI Architecture
summary
Combining Iceberg with simple Python functions run in the cloud turns S3 from a dumb blob store into tables on S3 that support pushdowns and time travel, plus branching/versioning like git for data. The concrete claim: this stack cuts bandwidth and deployment friction so teams prototype and deploy the same code. A real deploy at Mediaset (20M users, 10B views, 76 PB/year) runs ~100 pipelines on day one and thousands of runs after 20 weeks.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:09AI production challenges
- 5:21S3 limits for AI pipelines
- 8:14Git for data versioning
- 10:43Tables on S3 and cloud functions
- 13:55AI pipeline syntax in practice
- 16:45Myths on AI infrastructure
- 17:47Mediaset adoption case study
- 19:53Branching and versioning essentials
- 20:56Good APIs over tools
- 21:56AI with code-based platforms
- 22:40MCP servers community
- 23:02Call to action
related talks
50mDesigning a scalable data platform based on Apache Spark/Iceberg
32m7 steps to build-your-own data pipeline - for day 1 of your startup
29mSimplifying data flows
30mArtificial Intelligence More Like Artificial Stupidity
38mBuilding Practical, Cost-Efficient GenAI Solutions Using Serverless
01h 01mCurrent Methods for Building Data Pipelines
18mAI Should Not Replace Well-Built Data Fundamentals
11mBreaking Data Silos: Enterprise Data Management