Real-Time Event Processing and Handling Stateful Applications on Apache Mesos
summary
GE Digital describes running thousands of developers and ~30,000 apps on a Mesos-backed Predix platform that ingests extreme edge data (a GE jet engine ~1 TB/day). The team builds a Cloud Foundry marketplace that provisions stateful services by mapping CF broker → Mesos Cosmos calls to Marathon/Marathon-lifecycles, implements a ~100–250 line Scala broker (to be open sourced), runs Portworx persistent volumes for Cassandra/Elasticsearch, and operates Spark Streaming and Heron/Storm for near‑real‑time and sub‑ms latency processing; demo provisioning and service‑key injection works though the live network is flaky.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 0:40Agenda
- 1:16IOT Platform
- 2:18GE
- 5:11Cloud Pressure
- 5:48Predicts Platform
- 7:37Cloud Foundry
- 8:50Why Mesos
- 9:47How do we bridge both worlds
- 10:55Demo
- 11:55Service Broker
- 13:11Cosmos
- 14:18Call Flow
- 16:07Persistent Volume
- 18:59Mesa Skies
- 19:32Jet Engines
- 21:25RealTime Architecture
- 22:31Challenges
- 25:22Demo 1 Marketplace
- 32:08Cassandra OSB Package
related talks
39mThe Sparkling Solution that Scales to IoT
25mAccelerating Data Processing and Consumption for Building Modern Apps at the Edge
12mEdward Hsu, Mesosphere | DockerCon 2018
39mDeploying Fast Data Pipelines
27mGetting the Most Out of Your Mesos
38mHow to run Real Time processing on Big Data
10mImpacting Business as it Happens
19mKeynote Panel Discussion: SMACK in the Enterprise