Debugging and Observability for Distributed Ray Applications
summary
Describes concrete debugging and observability features added to Ray: a memory view that shows object references and call sites, custom metrics recorded via a simple API and scraped by Prometheus/Grafana (a demo shows a 1 million ping count over three minutes), and a new per-node dashboard agents process that collects telemetry locally to eliminate head-node polling and reduce overhead. Shows a real memory-leak case where queued lightweight tasks keep object references alive and is fixed by throttling task submission; distributed tracing via OpenTelemetry is in active development.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:25Ray fundamentals
- 2:35Debugging challenges
- 5:26Debugging tools and techniques
- 11:52Managing ray objects
- 15:54System observability
- 16:48Monitoring architecture
- 20:30Roadmap and conclusion
related talks
15mFull-Stack Observability in Ray: Infrastructure Signals in Ray Dashboard
37mEvolving Ray Core: Scalability, Reliability, and Compiled Graphs
17mRay History Server: Observability for Ephemeral K8s Clusters
26mData Processing on Ray
12mWhere’s My Training Job? Ray & GPU Observability at Datadog
27mEvolving and Scaling Ray for the Next Generation of AI Workloads
28mA Deep Dive into Ray’s Scheduling Policy
21mRay Train on Anyscale: Observability and Reliability at Scale