Data Analysis in Parallel
summary
Shows how to take a Jupyter workflow and run three real tasks — aggregate stats, a top-20 lemmas histogram, and a sentiment pass using a serialized Keras h5 model with tokenizer — across four company CSVs (Amazon, Microsoft, Netflix, Google). Uses IPyParallel (launch with ipcluster -n 4), compares sequential ~65s to roughly half that time in parallel, explains dependency decorators, DV direct view vs load‑balanced, map vs map_async, remote dv.sync_imports, and debugging via exceptions or per-engine log files.
AI-written summary · about
chapters (11)
- 0:00Parallel data analysis intro
- 1:51Why parallelism matters
- 5:44Analysis tasks overview
- 10:22Demo setup and data exploration
- 12:20Sequential execution
- 23:47Parallel with IPython
- 31:11Performance analysis
- 36:41Debugging parallel
- 48:07Running on remote machines
- 49:56Advanced parallel options
- 51:57Q&A
related talks
01h 01mParallel Python Analyzing Large Data Sets
01h 03mDask: Next Steps in Parallel Python
01h 24mData Science at Scale with Dask
01h 23mData Science at Scale with Dask
01h 22mUsing Dask for Parallel Computing in Python
34mData Products, From 0 to data science pipeline
29mScaling Data Workloads with Cluster Managers
03h 07mData of Unusual Size: Interactive Visualization