It's Abstractions All the Way Down...
summary
Argues that “all non-trivial abstractions are leaky” and you must be able to inspect one layer below to debug; shows concrete leaks: Power BI direct query runs 13 serial queries (3s each → 39s) versus parallel, and a Spark aggregation over 5 billion rows produces 11 million outputs with ~30 nondeterministic ties due to floating‑point grouping; prescribes concrete fixes: expose ETL, docs and ownership, build data products with code & wiki, and adopt the 80-16-4 user model.
AI-written summary · about
chapters (13)
- 0:00Introduction and context
- 5:17Defining leaky abstractions
- 7:20Organizational abstractions
- 10:44Constraints on complexity
- 13:01Best practices for tools
- 17:11Separation of concerns
- 18:35Mental models and metaphors
- 24:14Floating point math issues
- 29:21Addressing organizational silos
- 34:24Evolution of data science
- 39:17The 80/16/4 user framework
- 44:18Key takeaways and conclusion
- 45:33Q&A and closing remarks
related talks
25mThe Intro to Abstraction I Wish I'd Received
45mAbstraction Considered Harmful (and other opinions about writing maintainable code)
37mPitfalls and pro-tips for effective and transparent BI tools data science & applications
46mThe Right Kind of Abstraction
48mBuild Abstractions Not Illusions
46mPlatforms: Build Abstractions, not Illusions
37mHiding The Lead
38mData Secrets from a Platform Engineer