PyData in Production: Lesson Learned from Various PyData Deployment Strategies
summary
Describes concrete deployments of the PyData stack: Pandas and sklearn models run behind a Java service layer using Apache Thrift as an RPC shim, interactive analytics microservices built as a pool of Python workers, and cluster jobs moved to data with PySpark plus distributed conda environments via an open‑sourced spylon helper. Claims: hundreds of terabytes remain on HDFS, this approach works and speeds delivery, but incurs analytic scope creep, double implementation and serialization overhead that sometimes forces Scala rewrites.
AI-written summary · about
chapters (8)
- 0:00PyData in production
- 3:33Two teams different worlds
- 6:05Interactive analytics microservices
- 14:30Batch processing limits
- 19:50Streaming MapReduce challenges
- 23:24Apache Spark for Python
- 29:47Future of data platforms
- 30:29Q&A
related talks
17mProductionizing Data Science Workloads
42mScalable Data Science in Python and R on Apache Spark
29mBuilding scalable and robust Data Science Solutions
32mDeploying Python models to production
29mData Science Best Practices : From Proof of Concepts to Production
01h 23mData Science at Scale with Dask
24mSmart approach in development and deployment process for ML models
22mBringing Deep Learning into the Big Data Analytics