Measurement and Analysis for Sustainable Service Delivery
summary
A development team standardizes metric-driven experiments with predefined goals, risk checks, and required events, then runs A/B tests using Firebase and monitors results with Redash dashboards plus Kibana/Atlas visualizations. Removing an interstitial increased per-user article views and in-app pageviews but reduced custom-tab (external browser) opens by ~20–30%, so they ran a three‑arm follow-up adding prominent in-article links to recover browser traffic, achieved larger gains (more per-user reads and +3% PV), and proceeded with staged rollout while continuously monitoring dashboards.
AI-written summary · about
chapters (11)
- 0:00Sustainable service delivery intro
- 1:52Effect verification importance
- 5:25Verification plan design
- 8:56Case study and hypothesis
- 12:46Metrics and A/B tests
- 17:42Data collection and monitoring
- 19:31Analyzing test results
- 20:33Data-driven decisions
- 24:09Finalizing changes
- 25:11Data-driven culture
- 26:40Summary
related talks
55mHow to Improve Your Game's Retention & Monetization Through Data-Driven Experimentation
46mTweaking and AB Testing Your App on the Fly
52mAB-tests, lies, damned lies, and statistics.
21mOptimize your app's revenue with Firebase
30mMeasurement in a Privacy First World
01h 15mExperimentation at scale: Fuel your application’s growth through A/B testing
52mBuilding an A/B Testing Framework that works for Developers
29mLeverage Firebase to Optimize Your App’s Monetization Strategy