Smarter Search With Spark-Solr
summary
Open-source spark-solr library provides a Solr-backed RDD/DataFrame so Spark reads and writes directly to Solr, letting teams store all data in Solr (no HDFS/S3) and run batch jobs, ML and faceted exploration in the same stack. Example: k-means run from Spark immediately faceted in Solr reveals one cluster with ~90% of documents, exposing data imbalance. Batch-computed item similarities or SVD recommendations are written back to Solr and served with an interpolated mixed-mode recommender parameter for quick A/B tuning.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:39Cold Start
- 3:41Overview
- 5:34Practical Data Science with Spark and Solr
- 5:46Why does Spark need Solr?
- 8:49Solr Key Features
- 9:26Why Spark for Solr?
- 10:36Why do data engineering with Solr and Spark?
- 11:27Example workflow: Searchhub
- 14:00Searchhub: Initial Exploration
- 14:52Smarter Text Analysis in Spark
- 15:37Searchhub: Exploratory Data Science
- 18:06Recommender Systems with Spark and Solr
- 19:48Experimenting with mixed-mode Recommenders
- 21:09Resources
related talks
40mSmarter Search with Spark, Solr and Machine Learning
46mDiscovery & Analytics with Hadoop, Mahout & Solr
21mCase Law and ML on Spark
55mSpark Architecture
52mSolving Analytical Problems using Apache Spark
01h 35mDeep Dive into the Big Data Landscape
53mBig Data: from mammoth to elephant
34mEvolution of Slack Search