Retooling for a Smaller Data Era
summary
Argues that real progress is concrete: adoption of DuckDB and DataFusion as embeddable engines, a common memory format in Apache Arrow, and multi-backend APIs like Ibis cut the overhead that made “big data” expensive. Gives numbers: 2004 servers had one core vs modern 96‑core machines (dual sockets ≈384 threads), so terabytes often fit and query fast on a single node. Describes a built fast, scalable interactive data explorer for dataframes and tables and reports broad success in reuse and interoperability.
AI-written summary · about
chapters (7)
- 0:00Welcome
- 8:08Data processing landscape
- 14:09Big Data hidden costs
- 19:11Composable data systems
- 23:25Apache Arrow and execution engines
- 26:38Multi-engine data stacks
- 29:26Closing & Q&A
related talks
58mUnderstanding Big Data for Software Engineers
58mComposable Data Systems, the new frontier
36mEmbrace the chaos: Composable data systems with fewer asterisks
50mA Short Summary of the Last Decades of Data Management
51mMotherduck: Data Analytics in the Post Big-Data Era
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
37mKEYNOTE: Data Architecture Turned Upside Down
33mDataEngBytes 2023 - SYD-T1-01 - Ryan Boyd