~/panel

Panel Discussion: Fragmentation of the Batch Ecosystem in Kubernetes, Challenges and Solutions

on YouTube 53m
Panelat Kubernetes Batch + HPC Day North America 2022 · on Monday, 2022-10-24· [kubernetes, distributed-systems]

speakers

name:Abdullah Gharaibeh
links:none on file
name:Diana J. Arroyo
links:none on file
name:Wilfred Spiegelenburg
links:none on file
name:Daniel Milroy
links:none on file
name:Albin S
links:none on file

resources

video:https://youtu.be/xQBkBRJdMCs
slides:none on file

summary

Panelists compare concrete systems and limits: Armada, Unicorn, MCAD and Flux (qflux/fluence) build schedulers, queuing, quotas and hierarchical borrowing because etcd chokes at ~100k simultaneous job submissions and Kubernetes events lack ordering or delivery guarantees. Armada reports scaling two orders of magnitude beyond etcd by buffering, while the Flux operator instantiates a mini cluster inside Kubernetes to run tens‑to‑hundreds of thousands of tasks. Remaining gaps: gang scheduling fragmentation, multi‑cluster federation, multi‑tenant quotas and shared filesystem health.

AI-written summary · about

chapters (22)

  1. 0:00Batch workloads on Kubernetes
  2. 2:06Declarative vs imperative batch models
  3. 5:47Challenges in batch workloads
  4. 7:51Missing batch features vs YARN
  5. 10:00Kubernetes for batch workloads?
  6. 13:52Solutions for batch scheduling gaps
  7. 15:34Batch vs Kubernetes adaptation
  8. 18:11Power user needs vs batch efficiency
  9. 21:01Multi-cluster batch federation
  10. 25:34Mathematical accuracy in batch workloads
  11. 27:02UI and identity for batch systems
  12. 31:54Multi-tenancy in batch Kubernetes
  13. 34:01Scaling etcd and dependencies
  14. 36:26Collaboration with traditional schedulers
  15. 38:09Improving etcd for batch frameworks
  16. 41:49Educating users for batch submission
  17. 43:15User workload correlation in Kubernetes
  18. 45:22Storage challenges for batch workloads
  19. 46:24Defending etcd and tools
  20. 47:50Common APIs for batch workloads
  21. 50:18Forums for batch Kubernetes discussions
  22. 51:05Closing remarks

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.