Azure Cosmos DB Changefeed tales from the trenches
summary
Change feed pulls changes from 80+ collections (largest ~26M docs) into lightweight Azure Functions using the Cosmos DB trigger (batch size 2000) that write blobs to a Data Lake, with one Stream Analytics job per collection to deserialize into Azure SQL Managed Instance. Event Hubs proved unable to absorb the nightly burst (8–11pm, ~98% of traffic); duplicate deliveries and the need to delete lease documents to reset processing are recurring pain points, and the functions require a read/write key. App Service scales to P2V2 and up to eight instances on CPU/memory thresholds. Synapse Link is the planned alternative.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:17Agenda
- 2:04The problem to solve
- 3:22Changefeed Processor components
- 4:45Changefeed lifecycle
- 5:16Changefeed recap
- 6:22Solution 2
- 9:51Changefeed lightweight function
- 13:17Issues with Changefeed
- 15:30The upcoming Ideal solution
- 16:51Scaling Up / Scaling Out (For 90+ collections)
- 18:48Deserializing
- 19:54Governance
- 22:30Conclusion
- 23:51Questions
related talks
39mBuilding Event-Driven Systems at Scale with Cosmos DB
01h 01mServerless & Schemaless with Azure Functions and CosmosDB
21mBuild Scalable Inventory System with Azure Cosmos DB
47mWorking with data using Azure Functions
05mReal-world examples of real-time applications using change feed
52mEvent journal in Azure (no Cosmos inside)
02h 02mAzure Cosmos DB: Liftoff
01h 02mAzure Cosmos DB - Low Latency and High Availability at Planet Scale