Ten-year Journey to 10,000 Production Machines
summary
Claim of running well over 10,000 production machines, proven by a decade of re‑architectures: starts as a Chef-based MVP with a pixieboot loop, moves to a brittle DAG and microservices, then consolidates into Go with a client as agent model and tighter backend to avoid server bottlenecks. A proprietary, threaded datastore replaces the file system; a customer HA cold start exposes over 10 million job logs and is fixed by pruning and load optimizations. Lessons: transparency, fail‑fast, API stability, pragmatic scaling not overengineering.
AI-written summary · about
chapters (8)
- 0:00Start
- 6:00Greg rides the racks Pre-wired OpenStack but different every single time.
- 7:24SUSE trying to scale Crowbar Architectural limits are SPEED OF LIGHT limits
- 14:07Technicolor Bake Off Digital Rebar v2 doing TOO MUCH Digital Rebar 3 simplicity was critical
- 16:48Huh? Our CLI is an Agent? Adding Workflows and moving parallel operations outside of the server
- 20:35Bank 10+ integrations Conference room archaeology of ops integrations
- 24:06Goodbye open core When locking and threading gets serious
- 31:26As Code • Embrace Heterogeneous Systems • Incremental discovery
related talks
19mEngineering for the Long Game
33mFrom Prototype to Production: Making Agentic AI Systems Reliable
35m100 machines do 100s work in 1s
14mCreating tools to cross machines and networks
39mTrust Deterministic Execution to Scale & Simplify Your Systems
39mTop Performance Challenges in Distributed Architectures
35mBuilding Systems that Last
46mPracticalities of Productionizing Distributed Systems