Should I develop my own DS library? Maybe.
summary
Aetna decides to build an internal library called PI Connect that wires conda setup, Jupyter, multiple back ends, and Hive/Spark defaults into a simple API; it uses Python 3+ via sky to run modern PySpark on an older cluster and implements intelligent downcasting and schema inference so writes and appends just work. Usage grows with a 3x headcount ramp and supports ~24 million members; reported savings offset initial development costs.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:40Abstraction in data science
- 8:45In-house library motivation
- 13:30Pain points driving development
- 17:27Target audience
- 19:55Core features and design
- 25:17Libraries enabling abstraction
- 26:44Simplicity in complexity
- 28:54Trade-offs of abstraction
- 30:59Is your own library worth it?
- 32:06Q&A
related talks
33mYou Don’t Need Spark for That: Pythonic Data Lakehouse Workflows
06mOpen by design, open for data
32mDemocratizing (Py)Data
39mEnabling Data centric Teams
34mpydiverse pipedag: A library for data pipeline orchestration
44mKeynote
54mKey Challenges In the PyData Ecosystem and How we Can All Make a Difference
41mAccelerate Silicon Design with Jupyter Notebooks