Building FAST Data Solutions with DC/OS on Azure
summary
Demo shows a sensor-producer writing to a Kafka topic with partitions, consumers read partition batches and write offsets into Cassandra, and a lag-reader aggregates per-partition lag and posts it to Vamp workflow auto-scaling. Deployment uses DC/OS on Azure (ACS Engine/ARM templates) and Portworx pooled storage to avoid disk-move issues. Rule-based autoscaling triggers when aggregate lag >500 (scale up) and <100 (scale down); live demo scales consumers to ~10 and then back down, proving the end-to-end approach works.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:27What is FAST Data
- 6:06Outline
- 7:18What are containers
- 9:59Local Demo
- 16:12Orchestrators
- 17:08Stateful frameworks
- 17:46Cassandra
- 19:44Data Persistence
- 20:03Container Persistence
- 21:17Pooled Storage
- 21:40Challenges of attached disks
- 23:28pooled software solutions
- 24:17Portworks
- 25:13Demo Cassandra
- 25:33Deploying Cassandra
- 30:49The next day
- 31:35What is VAMP
- 33:42VAMP Demo
related talks
31mEvent-Driven Architecture - A successful strategy for a journey to the cloud
44mMulti Cloud Mesos
03h 02mAPAC Live Stream
42mBuild a Real-Time Event-Driven Java Solution in Azure
51mIntroduction to Azure Data Platform
35mArchitectures and Trends for Event Stream Processing on Azure with Open Source Software
44mApache Spark on Azure - solution (...)
50mAzure Data Factory patterns and best practices