Systems Thinking with Poisoned Systems
summary
Argues that observability AI learns patterns not systems and fails when training data is data poisoning or missing; gives concrete failures: AI hallucinated CLI flags, looped through wrong Python methods for Google Cloud’s new soft‑delete (seven‑day) retention until someone reads source tests to extract the real API and then generates working Python code; an AI flags Cassandra as “underperforming” despite normal 90% heap usage; an Elasticsearch cost cut from $12,000 to $4,000 succeeds by manual index compaction and snapshotting, showing the need for human-in-the-loop.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:03Poisoned systems and AI observability
- 4:54Data poisoning in AI systems
- 7:02AI bias and historical assumptions
- 8:31Dangers of deskilling expertise
- 13:58AI complexity and debugging
- 15:24AI limitations in Google Cloud
- 18:20Real-world AI biases
- 20:49Opacity in AI optimizations
- 22:15Elasticsearch cost optimization
- 24:02Human oversight in AI systems
- 25:55Resilience engineering
- 30:11Systems thinking evolution
- 32:19Systems automation reality
- 35:12Conclusion
related talks
39mBack to Basics: Cutting Through the AI-Generated Code Noise
29mWhy did I break production? Is AI the sickness or the cure?
50mPanel Discussion: What Can Go Wrong With AI in Testing?
44mToxic Trust: The Psychology of Automation Bias in Generative AI
25mWhen Your "New Teammate" Hallucinates: Observability as the Safety Net for AI Code
30mThe Robots Are Coming (But Not Yet), or, Don't Make Smart People Do Dumb Things
09mAI and Chaos Engr: Smarter Failure Testing for Resilient Systems
57mAI Applications Security Puzzle