Can't Touch This!: When Things go Wrong With an Appliance

on YouTube ↗24m

" summary

A single-VM GitHub Enterprise image on 2 cores/8 GB runs fine for ~50 developers but fails at tens of thousands; customers generate huge diagnostic tarballs and logs that are painful to move and parse. The team builds a parsed support bundle with collectd metrics, a forked git that reports to gitmon, failbot exceptions fed into Elasticsearch, and a web uploader esp-tools with remote extraction. The system proves useful: it surfaces memory/LDAP/IO causes and usually avoids direct SSH, and it helped catch silent disk corruption with MD5 round‑trips.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 7:08Generate a Diagnostics File
3. 11:49Exceptions Log
4. 12:15The Authentication Log
5. 16:03Diagnostics Tab

" related talks

• 30mStupid Ideas For Many ComputersAja Hammerly · LA RubyConf 2015
• 34mRejectConfVarious Presenters · GoRuCo 2009
• 48mMind of a HackerRyan Davis · LA RubyConf 2015
• 46mThe Mess We're InJoe Armstrong · Strange Loop 2014
• 35mPEACE: Programming Expertly Amid Corporations & EnterprisesSteven Haddox · Wicked Good Ruby 2013
• 35mDevOps Days MTL 2019- Yannick BrosseauYannick Brosseau · DevOpsDays Montreal 2019
• 28mKeynote: Stupid Ideas for Many ComputersAja Hammerly · RubyConf 2015
• 01h 08mAutomated Dynamic Fireware Analysis At Scale: A Case Study on Embedded Web InterfacesAndrei Costin · Black Hat Asia 2016