Rethinking Testing for MCP Servers
summary
Built a BrowserStack documentation MCP server and tests show two LLM clients behave differently: one model ignores the tool entirely, another uses it but times out when prerequisites (start browser) are missing, causing 60s hangs. Proposes a practical, deterministic server-side testing model that fits CI: treat tool descriptions as contract testing, run deliberate out‑of‑order calls as sequence testing, and verify protocol and business failures under error channeling so the LLM gets actionable signals.
AI-written summary · about
chapters (18)
- 0:00MCP server testing challenges
- 1:30New testing approach needed
- 3:38MCP server descriptions and LLMs
- 5:24REST APIs vs MCP servers
- 6:27LLM behavior with MCP servers
- 8:11Client expectations shift
- 9:57Current MCP testing landscape
- 11:22Three-layer testing model
- 12:05Contract testing for MCP
- 15:52Practical contract testing
- 17:58Sequence testing workflows
- 18:39Sequence testing failure example
- 20:21Sequence testing pseudo-code
- 21:25Error channel testing
- 23:11Handling error responses
- 24:37Error testing pseudo-code
- 25:20MCP testing strategies summary
- 26:02Q&A
related talks
01h 29mWorkshop: Build Your Testing Sidekick- Custom Tools with Model Context Protocol
55mYour MCP Server is Bad (and you should feel bad)
29mMCPs are Boring (or: Why we are losing the Sparkle of LLMs)
25mAre MCP Servers Dead?
27mYour MCP Server is Bad (and You Should Feel Bad)
12mPrompting LLMs to use Tools and Resources with MCP
02h 49mMCP Servers: Best Practices, Design Choices and Their Consequences
31mHow to build an MCP server for Vue?