Reliably forecasting time-series in real-time
summary
Claims an algorithm that mixes seasonal SARIMA/Prophet-style fits with a non‑seasonal strategy that finds linear sub-windows and takes a weighted median of slopes and intercepts to be robust to level shifts and spikes. Implements algebraic matrixized sums in NumPy to run tens of thousands of linear regressions in milliseconds, and rewrites costly rolling quantiles in Cython, cutting end-to-end query latency from ~1.2s to ~1.0s in production for monitoring trillions of points per day.
AI-written summary · about
chapters (22)
- 0:00Real-time forecasting intro
- 1:11DataDog monitoring platform
- 2:13Why forecasting matters for alerts
- 3:57Capacity planning with windows
- 4:37Business metrics forecasting
- 5:41Requirements for adaptive forecasting
- 7:05Avoiding false positives
- 8:29Speed and scalability needs
- 9:32Evaluating forecasting models
- 11:40Testing Prophet on data
- 13:27ML limitations for forecasting
- 17:00Custom forecasting algorithm
- 18:45Linear regression for thresholds
- 19:47Detecting trend sub-windows
- 21:36Weighted median for slope
- 23:03Optimizing regression with NumPy
- 26:59Vectorized computation gains
- 28:26Handling complex with Cython
- 29:54Performance in production
- 30:16Key takeaways
- 33:55Conclusion
- 34:24Q&A
related talks
42mRobust Automated Forecasting in Python and R
01h 43m(Serious) Time for Time Series
40mMulti-Series Forecasting at Scale with StatsForecast
54mTime Series forecasting with NumPyro
42mRobust Automated Forecasting in Python and R
29mTime Series Forecasting Techniques in Python
01h 40mApplied Time Series Econometrics in Python and R
26mTime Series Analysis in Python: Easy Tools for Scientific Insight