From Spark to Elasticsearch and Back
summary
Describes a working pipeline that trains on Spark and serves via Elasticsearch, claiming 300 billion recommendations a month and reach to 800 million users, with an online stack handling 35k requests per second at ~30 ms. Explains hourly jobs that produce ready datasets, a five‑component framework with automated deployment gates and offline evaluation, and an Elasticsearch plugin that scores embeddings natively. Concrete performance numbers, tooling choices, and a published engineering paper make the implementation and results verifiable.
AI-written summary · about
chapters (13)
- 0:00Start
- 1:46train's NLP Engine
- 4:11dictive Models
- 5:03Machine Learning Challenge
- 8:03at Makes Preparing Datasets a Hard Work?
- 10:47Need for an Machine Learning Framework
- 12:17Machine Learning Framework on Top of Spark
- 13:04omatic Data Collection
- 13:44ine Model Evaluation
- 15:38Framework for both Research and Production
- 17:37tributed Search Engines - What Can They Do?
- 19:35ware Search Engines Efficient?
- 25:07Takeaways
related talks
01h 01mBest Practices With Elasticsearch
36mRecommendations at scale with Elastic Search
35mIntegrating a Spark ML Pipeline into a High-Load Low-Latency Recommendation Service
45mReal-time, Streaming Advanced Analytics....
01h 35mDeep Dive into the Big Data Landscape
01h 00mElasticsearch: you know, for more than search
44mData Science at Scale with Apache Spark
21mCase Law and ML on Spark