Embracing the Chaos: Investigating Production Madness
summary
Debugging process shows concrete wins: query logs in Snowflake expose slow devices tied to a USB barcode scanner froze GC path in AOSP (bug present since 2013, fixed in Android 16), traced with Perfetto traces and reproduced locally; CPU and thermal misconfigurations (fake thermal probes and reversed YAML thresholds) throttle cores; a memory bandwidth governor disabled in release builds makes a 5GB-per-CPU copy task jump from 4.2s vs 29s; documented steps, breadcrumbs, and targeted fixes resolve root causes rather than masking them.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:52Android app latency
- 8:56AOSP barcode scanner GC freeze
- 12:50Dangers of crash fixes
- 17:03Structured docs issues
- 21:39Android OOM errors
- 27:12Performance regressions after OS upgrades
- 39:55Debug vs release performance
- 40:38Key takeaways
related talks
35mIn Search of Performance: the Subtle Art of Android Profiling
37mDebugging Android Devices in the Field
38mA Busy Android App Developers' Guide to Perfetto
44mPAPA: Performance of Android Production Applications
40mDebugging Android Performance at Scale: Lessons from Building Datadog's Mobile Profiler
36mDAAS: Debugging as a skill!
44mCoordinating Space and Time
33mUI profiling in examples. Where is my bottleneck?