Signatures, patterns and trends: Timeseries data mining @ Etsy
summary
Etsy processes about one million metrics in real time and builds Kale v1 as two components, Skyline (anomaly detection) and Oculus (similarity search), but the stack—Redis, Elasticsearch, Flask, Sinatra and an Elasticsearch DTW plugin—proves too heavy to adopt. Fingerprinting with a shape description alphabet plus fast dynamic time warping works for similarity, but point‑outlier methods generate many false alarms. Kale 2.0 reimplements core pieces as a Java library using ReactiveX, wavelets/FFT, and tests like generalized ESD, Kolmogorov–Smirnov and Mann–Whitney U.
AI-written summary · about
chapters (9)
- 0:00Introduction and Etsy background
- 3:17Kale version one overview
- 7:10Challenges with anomaly detection
- 13:32Limitations of point outlier detection
- 19:31Kale version two approach
- 23:26Design choices for Kale 2
- 26:56Reactive X and pipeline design
- 29:45Live demo of Kale 2
- 33:00Lessons learned and summary
related talks
55mSearching for Time Series Patterns at Big Data Scale
31mFinding Needles in a Growing Haystack
52mDevil in the Haystack
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
45mData as culture as software
01h 00mElasticsearch: you know, for more than search
25mGraph-based ML Anomaly Detection and Insights for Envoy Systems
50mOur experience at Etsy com with building “Big Data” analytic capabilitie