Toxic Trust: The Psychology of Automation Bias in Generative AI
summary
Shows automation bias causes real failures: ChatGPT-generated abstract gets accepted then ridiculed; GitHub Copilot suggests bubble sort and hides infinite-recursion fixes; GPS leads drivers into water; a Dutch welfare model and an AMS-style classifier produce systemic discrimination; a hospital dosing automation dispenses 38.5 pills instead of 160 mg and nearly kills a child. Recommends concrete fixes: four-eyes principle, post model-level error metrics prominently, prefer systems that offer information not recommendations, and warns training often increases user overconfidence.
AI-written summary · about
chapters (13)
- 0:00🚀 Build Stuff Conference – Get Your Tickets!
- 0:30Introduction: The “AI-Generated Abstract” Story
- 4:00What is Automation Bias? (And Why It Matters)
- 7:20Cognitive Biases Explained (Confirmation, Logical Fallacies, etc.)
- 11:30Model Bias vs Automation Bias (Key Differences)
- 15:10Real-World Case Studies: When AI Goes Wrong
- 20:40Errors of Commission vs Errors of Omission
- 24:10Everyday Examples: GPS, Google, and Developer Tools
- 28:30AI Coding Pitfalls: Copilot & Misleading Suggestions
- 31:40Why We Trust AI: Psychology, Stress & Complexity
- 34:20What Doesn’t Work: Training, Accuracy & “Human in the Loop”
- 36:00What Works: Metrics, Reviews & Better System Design
- 37:00Final Takeaway: Build AI Responsibly & Think Critically
related talks
57mToxic Trust: The Psychology of Automation Bias in Generative AI
52mSome pitfalls in AI
23mOpening up the box: the complexity of human interaction with AI systems
12mThe Comfort Trap: Getting Too Used to AI
23mHuman-Driven Intelligence: Where AI & Automation Fall Short
01h 07mUnpoisoned Fruit: Seeding Trust into a Growing World of Algorithmic Warfare
49mMagic of Automation and Everyday Chores
34mScaling Trust, Not Automation - Rethinking Quality for the AI Era