Enhancing Data Ingestion and Enrichment at Aware with Azure Cosmos DB

on YouTube ↗10m

" summary

Aware uses Azure Cosmos DB with KEDA autoscalers and change feeds to ingest and enrich message streams, capturing data at ingest, triggering transforms and ML enrichments, and storing results in Cosmos collections. Partition key choice and hot-partition fixes drive schema changes; logging RU per query and a 10% reservations rule prevent wasted capacity. Throttles (HTTP 429) lead to workload smoothing rather than immediate cluster scale-up. Data store doubles with no read/write latency increase and moves from dozens of per-customer DBs to a handful of global instances.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 0:27Business problem overview
3. 1:31System architecture
4. 2:57Why Cosmos DB
5. 3:59Ingestion pipeline switch
6. 4:42Partitioning strategies
7. 5:47Query tuning
8. 6:08Change feeds
9. 7:33Reservations and autoscaling
10. 8:41Monitoring and logging
11. 9:23Migration results

" related talks

• 01h 03mData Modeling and Partitioning in Azure Cosmos DBMark Brown · NDC Oslo 2021
• 32mAzure Cosmos DB Performance Tuning - Sonal Pathak & Amresh KrishanmurthySonal Pathak, Amresh Krishanmurthy · Azure Cosmos DB Conf 2022
• 48mPerformance tuning for Azure Cosmos DBHasan Savran · NDC Oslo 2024
• 01h 02mAzure Cosmos DB - Low Latency and High Availability at Planet ScaleKevin Pilch · NDC London 2024
• 18mIntelligent Resource Optimization: Mastering Azure Cosmos DB for Enterprise WorkloadsSrinivas Reddy Mosali · Azure Cosmos DB Conf 2025
• 33mDriver for Data-Driven Change - Azure Cosmos DB Performance, Security & Cost Optimization Done RightKristoffer Axelsson · Azure Cosmos DB Conf 2022
• 01h 00mCost cutting strategies for Azure Cosmos DBHasan Savran · NDC Porto 2022
• 23mServing Customer Data with Azure Cosmos DBHeber Ng · Azure Cosmos DB Conf 2022