Introducting Dask-Gateway: Dask clusters as a service
summary
Dask-Gateway provides a secure, multi‑tenant server that lets users launch Dask clusters via a REST API and central admin policies, supporting backends like Kubernetes, YARN and HPC job queues (Slurm/PBS). Admins lock options, set resource limits and idle timeouts; adaptive scaling and proxied dashboards work in demos, and the scheduler proxy uses TLS SNI to route encrypted TCP. The proxy is written in Go and bundled with the Python wheel, tests run with pytest and Docker. Dask-Gateway provides a secure, multi-tenant server scheduler proxy uses TLS SNI Go proxy bundled with the wheel
AI-written summary · about
chapters (16)
- 0:00Dask and talk intro
- 1:02How Dask works
- 2:26Dask cluster challenges
- 8:26Introducing Dask-Gateway
- 11:27Live demo of Gateway
- 14:33Long-running clusters
- 15:56Locking user options
- 17:22Resource limits and shutdown
- 18:26Configuring Gateway with Traitlets
- 21:55Multiple cluster backends
- 24:53Secure scheduler proxy
- 27:23Testing Gateway backends
- 29:32Key features of Gateway
- 30:35Documentation and community resources
- 31:44Q&A part 1
- 36:03Q&A part 2
