Bulk read and update millions of items in seconds with the Azure Cosmos DB Spark Connector!

on YouTube ↗12m

" summary

Demonstrates using the Azure Cosmos DB Spark Connector to perform parallel read many API calls and achieve a million point reads in about a second from a Databricks notebook, then run an item bulk update that uses read many + patch to update documents (demo run ~4 minutes on low provisioned RU/s). Shows using a Max item count of 1000 to batch ID/partition-key tuples, explains when queries vs point reads are chosen, and provides sample code in the connector repo.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 1:50Read Many API solution
3. 3:37Scaling to millions of docs
4. 5:24Demo setup in Azure Databricks
5. 7:08Limitations of traditional queries
6. 8:53Read Many API in Spark
7. 9:56Demo results and performance
8. 10:39Bulk updates with Item Bulk Update API

" related talks

• 33mDriver for Data-Driven Change - Azure Cosmos DB Performance, Security & Cost Optimization Done RightKristoffer Axelsson · Azure Cosmos DB Conf 2022
• 45mGetting the Most from NoSQL in the Cloud with Azure Cosmos DBJosh Lane · Oredev 2017
• 03h 30mAmericas Live StreamGuillermo Rauch, Scott Hanselman, Kirill Gavrylyuk, Deborah Chen, Estefani Arroyo, Natasha Dias Over, Brandy Umdenstock · Azure Cosmos DB Conf 2022
• 01h 03mAzure Cosmos DB - The Best NoSQL Database You're Probably Not Using (Yet)Josh Lane · NDC Oslo 2017
• 28mAccelerating Real-Time Analytics with Cosmos DB and GPU-Enhanced Serverless Apache SparkBrian Benz · Azure Cosmos DB Conf 2025
• 27mAzure Cosmos DB Changefeed tales from the trenchesRegis Baccaro · Azure Cosmos DB Conf 2022
• 53mOptimizing your Azure SQL Database workloadsSanjay Mishra, Denzil Ribeiro, Dimitri Furman · SQLBits 2020
• 50mHow I have improved the Database performance from 5 seconds to 1 millisecondManish · AzConf 2020