One AI, two AI, red AI, blue AI: Impacts of Aggregation on Model Diversity and Consumer Utility
summary
Analyzes how noisy aggregation of multiple models changes user welfare and firm incentives, formalizes aggregation with a Bradley-Terry/softmax choice model, and proves that standard win rate rewards homogenization while sometimes reducing consumer utility; proposes weighted winrate (value × selection probability) which provably encourages specialization and raises welfare. Shows nonmonotonic regimes where improving a model can harm users (empirically up to ~80% of models at mid temperatures on MT-Bench-101), and reports simulations validating the theory.
AI-written summary · about
chapters (9)
- 0:00Model diversity and utility
- 1:45Model aggregation and choice
- 9:50User welfare and new models
- 14:47Model replacement and utility
- 19:04Incentivizing creators with win rates
- 23:59Model replacement incentives
- 33:32Experimental results on markets
- 38:29Future directions
- 39:11Summary
related talks
46m(Improving) Muddled Information
26mHow to develop useful and unbiased AI
45mAGT and Machine Learning
31mFrom Single AI to Multi AI Systems
49mAn AI with an Agenda
20mWhat can AI do for me?: evaluating machine learning interpretations in cooperative play
01h 11mA Theory of Multi-objective Machine Learning
45mTutorial: Incentives for Collaborative Learning and Data Sharing, Part I