Making Sense of Streaming Sensor Data: How Uber Detects On-trip Car Crashes
summary
Pipeline joins GPS, accelerometer and gyroscope streams via Kafka/Avro and Flink to run a TensorFlow RNN deployed through Michelangelo, processing under 10 TB/day of sensor data and reducing it to gigabytes for model input. Crash signal is an accelerometer >3g plus a long GPS stop; engineers condense terabytes to gigabytes with session windows and stateful joins. System sends ride-check alerts and routes events to an operations team; iterations move from post-trip to hybrid on-trip scoring.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:19RideCheck Announcement USA TODAY
- 4:44Building a Model
- 5:26Why Flink?
- 6:23Uber Infrastructure: Schema'd Kafka
- 7:13Uber Infrastructure: M3 Metrics
- 8:14Sensor Data at Uber
- 8:54Joining TBs of Sensor Streams
- 9:16Condensing Prior to Trip Joins/Aggregations
- 10:07A Modular Post-Trip Crash Detection Topology
- 10:43The Power of Flink: Joining by Trip ID
- 11:57Platformizing: When Use Cases Diverge
- 13:00Platformizing: Demand for Sensor Data
- 13:27Adding Sensor Embeddings to the Model
- 14:44Sensor Trip Aggregation
- 15:49Consolidation
- 17:32On-Trip Crash Detection
- 17:47The Future
related talks
40mBuilding Real-Time Applications: Cyclist Crash Detection
38mSampling and Modelling the World in Real-Time for Mobility
22mBuilding a movement data and analytics platform to de-stress customers & save lives
30mCrash detector
40mSmart cars of tomorrow: real-time driving patterns
19mThe Data Science behind Cruise’s Robotaxi Fleet
37mStreaming your shared ride
34mData Drives Business