Journey to consolidation
summary
Siri migrates its runtime from heavy virtualization and filers onto Mesos and HDFS so a tiny operations team can run global real time ML at scale; binaries launch on Mesos, models and staged data live on HDFS, and orchestration shifts to YARN for Hadoop integration. The system intentionally tolerates hotspots by keeping 50+ replicas of model data, eliminates expensive filers, and reports full migration to HDFS across data centers by year end.
AI-written summary · about
related talks
33mKubernetizing Big Data and ML Workloads at Uber
27mML Platform Journey
53mHow We Built a Highly Scalable Machine Learning Platform Using Apache Mesos
36mLessons Learned Running IBM Watson on Mesos
16mFrom Self-Managed Ray to Anyscale Services: Migrating ML Serving
31mRunning Machine Learning Analytics On Traces
44mBuilding Multi-Tenant Compute Systems in the Enterprise
21mML Infrastructure for Autonomous Vehicles