Applying Geospatial Analytics at a Massive Scale using Kafka, Spark and Elasticsearch on DC/OS
summary
Architecture uses Kafka, Spark (streaming and batch), Elasticsearch and DC/OS to ingest and analyze at millions of events per second, rearchitecting a monolith into microservices and edge sources with Kafka topics and Spark jobs writing to ES. A 30-node cluster deploys in about ten minutes and demos run from a Play web app backed by ES aggregations and space-time cubes. Example scaling used roughly ten Kafka brokers to hit those throughputs. Stateful streaming reliability remains unresolved so recurring batch jobs via Metronome run every ~30 seconds. The code baseline lives in the DCOs IOT demo repo.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 3:46Traditional approach
- 5:02DCOS
- 6:23Trinity
- 7:06Project Trinity
- 12:03Examples
- 12:49Temporal Operators
- 14:06Aggregate Points
- 15:10Aggregate Bins
- 16:14Demo
- 18:37Deployment Portability
- 21:18Deploy across different environments
- 23:38Installing DCOS
- 25:21Installing Kafka
- 25:40Installing Elasticsearch
- 26:54Kafka Source
- 27:57Map Interface Demo
- 30:22Simulations
- 32:17Challenges
- 33:21Open Source Extensions
- 34:06Autoscaling
- 36:04Metrics
- 37:46Stateful Processing
- 39:40Batch Analysis
- 40:25Recurring Analytics
related talks
39mThe Sparkling Solution that Scales to IoT
19mKeynote Panel Discussion: SMACK in the Enterprise
39mDeploying Fast Data Pipelines
12mEdward Hsu, Mesosphere | DockerCon 2018
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
46mKafka Streams at the Big Data Crossroads at Lightning Speed
39mAnalysing Public Kafka Data from NASA Satellites
02h 18mDistributed Systems: In the Daily Life of the Data Engineer