Observability for Dask in Production
summary
Shows concrete debugging wins: centralized logging plus a topic-based events stream, a native Prometheus plugin, and performance reports that capture realtime dashboards. Traces are skipped but a built-in statistical profiler and dataframe analyze aid root cause work. Examples include a P2P shuffle failure correlated to a no space left on device error, using metrics to choose m6i over burstable instances, and detecting a surprising 3x regression from a shuffle rewrite. Tools and alerts are practical and reproducible.
AI-written summary · about
chapters (14)
- 0:00Dask overview and use cases
- 4:19Live Dask cluster demo
- 6:51What is observability and why it matters
- 8:55Logging best practices
- 11:05Centralized Dask logs
- 13:56Dask events explained
- 17:33Metrics for observability
- 23:39Metrics use cases
- 26:32Data-driven Dask design
- 28:36Dask diagnostic tools
- 30:01Performance reports
- 31:07Profiling Dask workloads
- 32:50DataFrame performance insights
- 34:16Summary
related talks
26mObservabilidad en Python para principiantes
40mDemystifying Observability and Monitoring
28mResolving Outages Faster With Better Debugging Strategies
21mDebugging at Scale—Going from Single Box to Production
58mObserving Applications in Production
47mObservability - Holistic Quality across Software Systems
53mUnmask hidden issues with Observability superpowers!
56mFull-stack observability and application monitoring with