Building analytical microservices powered by Jupyter Kernels
summary
Claims a scalable, multi-tenant, secure way to run Jupyter kernels off the notebook server by turning kernels into cluster applications so heavy Spark drivers land on workers instead of the gateway; uses Enterprise Gateway to launch remote kernels in YARN cluster mode with Kerberos-based user impersonation and per-user HDFS homes. Demo builds a Flask REST sentiment microservice over a Yelp dataset that starts/stops kernels via the gateway, runs PySpark jobs to compute sentiment on demand, and shows the approach working end-to-end.
AI-written summary · about
chapters (6)
- 0:00Introductions and agenda
- 1:41Jupyter architecture
- 6:12Enterprise gateway concept
- 13:42Gateway architecture
- 15:26Building the microservice
- 22:06Demo and conclusion
related talks
45mData Science Apps: Beyond Notebooks
06mLightning Talk: Using Jupyter Notebooks To Gain Insight Of Your Cluster
46mComposable microservices for streaming analytics
24mJupyter Notebooks for Ops
50mBeyond Interactive: Empowering Everyone with Jupyter Notebooks
27mAdding interactivity by extending Jupyter Notebook
49mExperimenting with Large Data Volumes Using Notebooks
51mReal world microservices: Starting-up from scratch!