A physicist's approach to Data Science
summary
A city-specific log linear multiplicative model predicts utilization from demand, supply and speed using simple logs and scikit learn, achieving about R² ≈ 0.9 and estimating exponents (demand ≈1, supply ≈0.6). Residuals become a residual-based monitoring signal that spots regressions and even a deployed bug. Adding a noisy systemic feature — the entropy of the 4D pickup/dropoff distribution — improves fit and makes trend shifts and operational experiments easier to detect in real time.
AI-written summary · about
chapters (13)
- 0:00Introduction to talk
- 1:09Automotive industry revolution
- 3:36Problem statement and KPIs
- 6:24Challenges in data analysis
- 9:14Experimentation difficulties
- 12:41Physicist's approach to modeling
- 15:49Building a simple utilization model
- 18:32Model insights and operational decisions
- 20:18Using model to denoise data
- 21:43Detecting system changes with residuals
- 23:06Improving the model with entropy
- 26:46Key takeaways and future work
- 27:50Model limitations and generalization
related talks
40mMachine Learning over Small Data: generating new variables with predictive power
34mHow to understand user behaviour using Multiple Linear Regression?
33mForecasting Vulnerability Sightings Under Data Scarcity: Modeling Sparse and Bursty Cyber Threat ...
32mLean Experimentation and Decision Science with Humans in Warehouses
15mInterpretable Machine Learning with R
51mPlaying Newton: Automatic Construction of Phenomenological, Data-Driven Theories and Models
26mSridhar Telidevara: Professor, Great Lakes Institute of Management at Cypher 2017 - Part 1/2
31mScaling up Lyft’s Growth via Causal ML and Bandits