Beyond 1000 BOSH Deployments
summary
Claims running 1,214 deployments on a single BOSH Director by using anynines data service framework with dedicated instances and three directors: over (runtime), under (service brokers, 1,200+ deployments) and utility (monitoring). Troubles include AWS GP2 IOPS limits on RDS, automatic snapshot and inode storms from BOSH task locks, and a tasks table with millions of rows that makes Postgres queries slow. Remedies: upstream BOSH fixes, switch from Prometheus VM exporter to Graphite plugin, add targeted alerts for IOPS, inodes and cert expiry, and automation to keep directors responsive.
AI-written summary · about
chapters (12)
- 0:00Introduction and setup
- 1:49Road to 1200 deployments
- 3:57BOSH director architecture
- 5:48Early issues with GP2 disks
- 8:40Daily snapshot tasks impact
- 10:05Fall of 2018 problems
- 11:53Disk full but not full issue
- 14:47Slow BOSH director during upgrade
- 16:56Database task table overload
- 18:44Monitoring essentials
- 22:35Lessons learned from 1200 deployments
- 25:28Q&A session
related talks
30mStreamlining the BOSH Experience
27mBOSH: The Road Ahead
17mDeployments We Can Believe In
26mWays to Monitor your BOSH Director
25mBOSH: The Simple Solution for Complex Systems
28mHow to Deploy CF and Other Bosh Releases on Bare-Metal Environment
24mOn-Demand Service Provisioning with BOSH 2.0
25mBOSH Bootstrapping