Scale out from the very beginning
summary
Implements a nightly distributed filesystem scanner over 1.5 PB storage that crawls ~60 million files and ~500k directories, produces CSV indexes pushed to MongoDB, and exposes results via a REST layer; uses dask distributed and dask dataframes for parallel scans and analysis, with Bokeh/Flask for views. Claims enough speedup to run routinely (reduced multi‑hour scans to practical nightly jobs) and delivers dedupe, usage metrics, test datasets and a simple table of contents.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:25Speaker background
- 2:55Early data challenges
- 6:49Distributed file scanner
- 11:49Metadata for scans
- 16:27Multi-core scan optimization
- 21:30Scaling with Dask
- 28:22Key takeaways and scaling strategies
- 29:48Data tagging guide
- 33:19Scan analysis with Dask
- 35:33User bottleneck solutions
- 37:12Q&A
related talks
30mGeoscience at Massive Scale
01h 17mScaling PyData with Dask
01h 03mDask: Next Steps in Parallel Python
30mBlazing fast I/O of data in the cloud with Daft Dataframes
40mScaling Python - Bank Edition
01h 26mDask tutorial
25mDeploying Python at Scale with Dask
30mDask ❤️ Xarray: Geoscience at Massive Scale