Building low latency gradient boosted tree predictors on distributed platform
summary
Describes building a Spark + XGBoost training and experimentation pipeline to handle large-scale transaction data and extensive validation, and packaging the resulting models as a custom embeddable predictor library for Ekata’s Identity Check Confidence Score. Focuses on delivering real-time, low-latency predictions by integrating the XGBoost-based models into production as a fast, deployable predictor component.
AI-written summary · about
chapters (24)
- 0:00Start
- 2:05Ekata - Who we are?
- 4:53Identity Check
- 6:39History of ML at Ekata
- 9:57Building low latency predictors
- 12:06Predictor library
- 13:40Building Gradient Boosted Trees
- 14:37Training on a distributed platform
- 15:59The machine learning lifecycle
- 17:27The real life machine learning lifecycle
- 18:23TODO for model deployment
- 19:58Model descriptor formats
- 20:53Feature extraction - before
- 21:55Our approach
- 24:42Feature Descriptor
- 26:18Adapt to other languages
- 26:56Normalizer
- 27:10Building the models
- 28:09Current technology stack
- 29:56Evaluation
- 33:47Reports
- 37:36Delivering the model
- 38:08Using the model
- 38:21Summary
related talks
41mStreams to the River: Powering an Online XGBoost Classifier with Kafka
28mDistributed XGBoost on Ray
01h 26mImplementing a highly scalable Stock prediction system with R, GemFire and Spring XD
22mBringing Deep Learning into the Big Data Analytics
16mStop Failures Before They Happen: Gradient Boosting
43mDeploying Large Spark Models to production #bbuzz
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
37mAutomated Machine Learning: Building a Pipeline