Supercharge Spark: Unleashing Big Data Potential with Milvus for RAG systems
summary
Concrete claims and runnable tools: Milvus integrates with Spark via a spark-milvus connector that supports iterative and bulk inserts (Parquet/JSON → S3/MinIO), builds indexes (≈15 types, DISK/GPU options), and exposes Prometheus/Grafana metrics for k8s. Reported performance is <20 ms on 100M+ vectors and architecture scales to 100 billion vectors on distributed Kubernetes. Examples include hybrid filtering, bulk S3 staging, and GPU vector search; code and docs live on GitHub and the connector automates index creation.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:33Vector databases benefits
- 8:12Vectors in Milvus
- 12:49Scaling vector search
- 15:59RAG systems basics
- 18:06RAG pipelines data processing
- 21:41Scaling RAG with Spark
- 25:52Live demo Spark-Milvus
- 28:00Spark-Milvus connector
- 32:14RAG evaluation metrics
- 37:26Monitoring RAG evaluation
- 38:31Q&A
related talks
37mMilvus: Scaling Vector Data Solutions with Gen AI
35mMilvus 2.0: Building the World's Most Advanced Open-source Vector Database
36mMilvus: Scaling Vector Data Solutions with Gen AI
26mThe Rise of Vector Databases - Lessons from the Milvus Community
26mUnleashing the Potential of Cloud Native Open Source Vector Databases
40mEvolution of Milvus Cloud-Scalable Vector Database
11mScaling R with Spark
39mAccelerating Data Science with RAPIDS