How to Build a Performance Testing Pipeline from Scratch
summary
A production pipeline is built that runs UI scenarios on physical devices (cloud-backed via Firebase Test Lab), records network with OKReplay, collects start/stop timings to CSV, and compares dev vs master using the Mann‑Whitney U test on medians to trigger premerge Slack alerts. It runs ~31 attempts per build, integrates with Jenkins and Espresso idling resources, blocks merges by coupling with functional tests, and surfaces regressions and improvements; it finds real issues but accumulates ~1% noisy alerts and needs ongoing engineering (roughly 2–3 people) to stabilize.
AI-written summary · about
chapters (14)
- 0:00Introduction and background
- 2:35Performance challenges at scale
- 5:44Need for a performance testing pipeline
- 8:14Statistical approach to performance testing
- 12:07System architecture overview
- 13:53Performance test job design
- 16:48Cloud vs local device testing
- 20:21Device-level performance testing
- 23:10Key principles for stable tests
- 26:36Backend API and data handling
- 28:22Developer experience and UI
- 32:36Debugging and future improvements
- 35:06Results and ongoing work
- 37:15Q&A session
related talks
40mAutomated Performance Testing In Preproduction
33mPerformance Testing in Agile and DevOps
23mDistributed performance testing at the push of a button
35mUsing the DevOps Pipeline to Generate Continuous, Valuable Insights
53mTransforming testing methodologies: From manual to robust strategies
41mContinuous Performance Testing in DevOps
39mCreating & Scaling Meaningful Performance Tests in CI
31mHow we can manage performance testing?