HHVM at Etsy: Harder, Better, Faster, Stronger
summary
Etsy targets a 20x scale on its API cluster and prototypes HHVM after a local 5x throughput win, then measures roughly 2x throughput on production traffic. Migration builds an RPM, hits CentOS6 libc and extension compatibility problems (memcache decrement bug, hashing bug), and a threaded setlocale race that mangles logs; repo-authoritative mode is skipped to keep 60 daily deploys. Tooling proves valuable: perf, flame graphs, built‑in debugger, sgrep/s-patch for large refactors, and gradual tee‑traffic ramp works.
AI-written summary · about
chapters (23)
- 0:00Introduction to Etsy and HHVM
- 1:32History of HipHop and HHVM
- 3:26Adoption and Compatibility
- 5:47Performance and Etsy's Need
- 7:37API Development at Etsy
- 10:56Hardware Challenges
- 14:10Early Prototype Phase
- 16:48Compilation Struggles
- 19:11Infrastructure Hurdles
- 21:16Experimentation Strategy
- 23:12API Cluster Benchmarking
- 25:40Teeing Traffic Experiment
- 28:33Production Traffic Results
- 30:32Correctness and Extension Bugs
- 32:17Ramping Up Traffic
- 34:07Thread-Safety Issues
- 37:45Successful Implementation
- 38:56Tooling: Flame Graphs
- 40:39Tooling: HHVM Debugger
- 42:50Tooling: S-Grip and S-Patch
- 46:29Tooling: Perf
- 48:27Lessons Learned
- 50:38The Future and QA
