Scala With Uber Data Science

on YouTube ↗50m

" summary

Uber builds a Scala statistics engine that runs on Spark, Hive and Vertica to do on‑demand A/B tests with stage rollouts, using delete-a-group jackknife variance estimation instead of naive sequential tests to correct for clustered trips. A centralized feature store (Pallet) plus Michelangelo serves features in Cassandra for online prediction and supports partitioned models by global/country/city with automated pruning. The cash payment experiment increases first‑time rider adoption enough that some markets default to cash, and the pipeline integrates Jupyter + Livy for scalable data prep.

AI-written summary · about

" chapters (6)

1. 0:00Introduction and background
2. 3:38Cash payments at Uber
3. 10:46Experimentation with Scala
4. 21:00Machine learning at Uber
5. 39:06Spark Magic for data prep
6. 43:53Q&A session

" related talks

• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 05mInterviewNimbus Goehausen · Scale By The Bay 2018
• 43mBuild a Modern, End-to-End, Big Data Scala Reference ApplicationEdgar Orendain · Scale By the Bay 2017
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler · GOTO Chicago 2016
• 04minterviewUmayah Abdennabi · Scale By The Bay 2019
• 24mScio data processing nirvana at SpotifyJulien Tournay, Bram Leenders · Scale By The Bay 2018
• 46mScala and the JVM as a Big Data PlatformDean Wampler · Scala World 2015