Combatting illegal fishing with Azure for less than £10/mth
summary
Claims a serverless pipeline that processes 1.5 billion messages per month for under £10/month of compute by using Azure Functions consumption plan, Azure Data Lake with parquet data lake storage, ONNX models and durable functions orchestration. Implements a proximity detector using arterial spatial bounding boxes to cut candidate pairs from ~300 million comparisons/sec, uses list preallocation, Span<T> and streaming IEnumerable to shrink memory, and continuous benchmarking via Application Insights — and the pipeline meets the cost and scale targets.
AI-written summary · about
chapters (27)
- 0:00Start
- 0:28Problem Space
- 4:44Processing Pipeline
- 5:21Phishing Alerts
- 6:01Organizational Landscape
- 6:53Organizational Constraints
- 7:26Migrate to the Cloud
- 7:43Cloud Evolution
- 8:43Commodity Computing
- 10:55Cost Comparison
- 13:07Compute Cost
- 14:48Containerized Hosting
- 15:11Serverless Scale
- 16:51Data Structure
- 17:47Vessels and Proximity Workloads
- 19:31Machine Learning and Onyx
- 20:33Experimental Method
- 21:48Proximity Algorithm
- 25:53Durable Functions
- 26:47Function Chaining
- 27:18Async Http Api
- 27:50Fan Out Pattern
- 32:02Benchmarking
- 32:50Constraints in Azure Functions
- 33:27What Factors Affect Performance in Dot Net
- 33:57Garbage Collection
- 43:04Main Takeaway Takeaways
related talks
30mProcess large-scale data and computational function in serverless manner
58mAzure Data Lake & Synapse Server less Architecture
01h 01mServerless Azure Data handling
21mBreaking out of the habit of bulk loading everything
40mData Lake House in the Cloud: The Azure Solutions
44mFlying High with Data Engineering in Microsoft Fabric
20mChoosing between Microsoft Fabric, Azure Synapse Analytics and Azure Data Factory
24mREDLake Azure - Enterprise Data Lake and Business Satellites