PixieDust make Jupyter Notebooks with Apache Spark Faster, Flexible, and Easier to use
summary
PixieDust claims to make Jupyter + Apache Spark faster and easier by doing concrete things: automated install of Python and Scala kernels, a Scala-Python variable bridge that moves variables between runtimes, realtime Spark job progress in-notebook, and auto-visualisation using Matplotlib/Seaborn/Bokeh/MapBox. The demo pulls a Twitter stream via a shipped jar, runs Spark Streaming with Watson Tone Analyzer (30s run yielding ~158 tweets), and produces interactive charts and PixieApps HTML UIs that actually work in the notebook.
AI-written summary · about
chapters (11)
- 0:00PixieDust introduction
- 3:11Data science stress
- 5:17Record systems orchestration
- 8:28Collaborative project story
- 11:45Jupyter Notebooks and Spark
- 13:53Why developers love PixieDust
- 16:27Live Twitter demo
- 23:32PixieDust visualization and Watson
- 26:45Executive usability and Pixie Apps
- 31:14Building Pixie Apps
- 35:11Orchestration benefits and conclusion
related talks
52mBlurring the line between Developer and Data scientist with PixieDust
51mSpotlight on Emerging Data Science Worskspaces and Spark as Development Levers
38mPySpark in Practice
31mBringing the Jewels of the Python World to Scala with Spark
20mIntroduction to Zeppelin Notebooks and PySpark 2 0
44mUsing Python with Hadoop to create production ready big data applications
43mPySpark in Practice
24mPySpark - Data processing in Python on top of Apache Spark.