Data Scientists are More Valuable than Hardware: The Design of Modin
summary
Claim: data scientists are more valuable than hardware, so Modin makes pandas code run at cluster scale without rewriting. Modin is a drop-in pandas replacement that lowers ~600 pandas APIs to a ~14-op internal algebra, falls back to pandas when an API is missing, and uses Ray for execution. It adds a simple with-context to connect to multiple Ray clusters from one notebook. Over 5,000 GitHub stars and native runtime parity show the design works in practice.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:52Data Science Landscape
- 5:50Modin
- 11:55Data Scientist Insights
- 12:23Data Scientist Tools
- 13:55Spreadsheet API
- 14:59Data Scientists dont care
- 16:31Cluster Connection
- 18:50Data Scientist Productivity
- 21:39Outro
related talks
21mScaling Interactive Data Science with Modin and Ray
29mLearning to Scale Data Science, Machine Learning, and Pandas with Ray and Modin
46mScaling Data Science with Modin
39mAccelerated Data Science Libraries
30m"Better dataframes"
01h 22mHigh-Performance Data Science at Scale with RAPIDS, Dask, and GPUs
39mAccelerating Data Science with RAPIDS
28mThe Coding Portion of Data Science