Discovery Found Everything: Your Agent Still Doesnt Know What Matters
summary
An AI service desk analyst agent runs in production, restricted to one incident type on a single service while ServiceNow performs nightly ServiceNow Discovery to keep the asset layer current. The claim: discovery finds what exists but not business meaning—service mapping, ownership, criticality and dependency behavior are missing. Example failure: the agent routes access approvals for the Vision system to stale owners or grants authority nobody should have. Recommendation: apply the Context Integrity Test (four questions) and gate what the agent is allowed to do.
AI-written summary · about
chapters (49)
- 0:00Start
- 0:04Session Opens - Discovery Found Everything, Your Agent Still Doesn't Know What Matters
- 0:35We Ask Whether Agents Reason Well, Not Whether Their Data Is Good
- 1:05Your System Knows What's Deployed - Not Who Owns It or What Breaks
- 1:35Speaker Intro - Angela Cutulenco, Senior Director at Toronto Transit Commission
- 2:06Twenty Years Ago: A CMDB Built by Hand in Microsoft Access
- 3:08Today: ServiceNow Discovery Doing in a Night What Took Months
- 3:41Leading an AI Service Desk Analyst Agent Implementation
- 4:11How They Chose That One Incident Type - the Reason for This Talk
- 5:12Almost Nobody Asks Whether the Information We Gave It Is Any Good
- 5:44Two Arguments: Your CMDB Is Accurate, but Accurate Isn't Trustworthy
- 6:46What Nightly Discovery Actually Finds
- 7:48The Asset Layer Is Often the Most Current Record in the Organisation
- 8:18The Problem: Solving the Part We Know How to Solve and Calling It Done
- 9:19Complete Is Not the Same as Trustworthy
- 10:54The Questions You Actually Ask at 3 AM
- 11:26A Server Can't Tell You a Rider Is Waiting on the Platform
- 11:56The CMDB Is Really Two Databases in the Same Tables
- 12:26Layer One: Service Mapping
- 12:58The Paratransit Scheduling System That Takes People to Dialysis
- 13:30Layer Two: Ownership
- 14:01The Reorganisation That Split One Team Into Three
- 14:33Layer Three: Criticality
- 15:03An Agent Reads Empty the Same Way It Reads Low
- 15:36Layer Four: Dependency Behaviour - the Most Important of All
- 16:06The Never-Fail Architecture That Recreates the Server at the DR Site
- 16:37All Four Layers Depend on a Human, and None Force Themselves Current
- 17:38Accurate About What Exists, Unreliable About What It Means
- 18:08So Why Has Nothing Terrible Happened Yet?
- 18:39What an Experienced Engineer Actually Does With That Record
- 19:39All of It Happens in About Four Seconds Inside a Person's Head
- 20:09They Don't Consider It Expertise, So Nobody Ever Wrote It Down
- 20:40Why the Access Database Worked - Nothing Entered It Without an Approved Change
- 21:12Discovery Took Over the Writing - and the Meaning Stopped
- 21:43Handing That Same Record to an Agent
- 22:44The Agent Is Not Making a Mistake - It Follows the Logic Correctly
- 23:16The Part That Concerns Her Most
- 23:48Automating the Work Removes the Knowledge With It
- 24:19What the Failure Actually Looks Like - Not the Dramatic Kind
- 25:20The Walkthrough: A User Loses Access to a Tier 1 System
- 26:22The Record Doesn't Say When That Owner's Name Was Entered
- 26:54Either Nobody Approves, or They Approve Access They No Longer Can
- 27:28Where Did the Agent Make a Mistake? It Didn't
- 27:58It Passed Every Test - Because We Tested How It Thinks
- 28:29Why Better Evaluations Won't Solve This
- 29:00"Fix the CMDB First" Is a Multi-Year Programme That Never Finishes
- 29:33Stop Asking How Good Your CMDB Is - Ask What the Agent Is Allowed to Do
- 30:06The Context Integrity Test and Its Four Questions
- 30:36The Real Question Isn't Whether the Agent Is Good Enough
related talks
16mAI Agents for Incident Investigation: The Good, The Bad, and The Ugly
25mAgents Are Not Users or Services: Securing AI Traffic with a New Identity Model
36mWhere Should the Agent Act? New Approach to Product Discovery for Agentic Systems
41mHow to Tell If Your Agent Used the Right Stuff
16mMulti-Agent Intelligence for Next-Gen Insights
30mAI Agents for Incident Management
19mAI Agents for Incidents: The Good, the Bad, and the Ugly
20mThe Three Ways AI Agents Fail. Here's How to Fix Yours.