SPONSOR WORKSHOP / Zhiyi Ma, Shagun Sodhani / Facebook
summary
Platform describes a runnable pipeline that hosts submitted model code on AWS SageMaker using torchserve and Docker then runs batch evaluations across datasets so inference metrics, compute and fairness/robustness perturbations are measured automatically; Dynabench exposes four tasks and currently six models on its leaderboard with a top dyna score of 43.60 and lets users submit via the open source Dynalab CLI to the evaluation cloud. Separately, MTRL and MTEnv provide plug‑and‑play multi‑task RL components and mt10/mt50 environments.
AI-written summary · about
chapters (16)
- 0:00Start
- 0:26Agenda
- 4:56What Do We Do with the Benchmarks
- 8:58Job Scheduler
- 11:44Evaluation Cloud
- 12:15Types of Data Sets on Evaluation Cloud
- 17:03Demo
- 19:11Change the Dataset Weight
- 20:25Dynalab
- 20:46How Does the Model Upload Process Work
- 21:50Test the Code
- 25:58Verify Response
- 26:45Multitask Reinforcement Learning
- 27:03Reinforcement Learning
- 29:22Why Should We Build Environments and Baselines for Multitasking
- 29:52Ease of Use
related talks
39mTrain ML models at scale with Amazon SageMaker Training
19mDeveloping large-scale machine learning algorithms on Amazon SageMaker
37mStreamlining Machine Learning Operations
34mLat Designing and Building Complex Machine Learning....
29mTeam Teardown: MLOps in the Cloud for IoT
35mCreating Your Own LLM Tuning Platform with Open Source Technologies
34mServing ML Models at Scale with Seldon and Kubeflow
33mDeploying and Scaling Spark ML and Tensorflow AI Models