Polars, the End of Spark?
summary
Compares Apache Spark and Polars for processing tens to hundreds of gigabytes of data, arguing Polars offers better performance, simpler APIs, and greater efficiency. Presents a direct benchmark-style comparison to make the case that Polars can replace Spark as a modern alternative for data processing.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 3:08What is Polars
- 5:33What is Spark
- 7:38Polars vs Spark theory
- 8:20Big data limits
- 9:43Practical experience
- 13:52Group buy results
- 15:17Broadcast join results
- 17:48Large join results
- 19:13Results summary
- 22:26Polars limitations
- 24:31Polars future tips
- 26:35Conclusion
- 29:07Q&A
related talks
32mPandas, Polars and the DataFrame Consortium.
30mIs the great dataframe showdown finally over? Enter: polars
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
58mFrom Pandas to Polars Upgrading Your Data Workflow
21mPolars: A highly optimized dataframe library
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
28mHow I used Polars to build built functime, a next gen ML forecasting library
30m"Better dataframes"