You are evaluating an MCP server before integrating it into a production agent. What criteria would you use to assess the quality of its tool definitions? Name at least five.
Free · 2 AI evals / day
An **MCP server** is a third-party process your agent connects to in order to get tools, files, and prompts. Treating it like a trusted library is the mistake. You vet it the way you vet any dependency before shipping it.
Start with the tool definitions the model actually sees. Each tool has a description and an input schema. The description must be accurate and specific, because the model uses it to decide which tool to pick. A vague or oddly long description is a warning sign: it may hide instructions aimed at the model.
Then check the schema. Tight schemas with required fields, enums, and bounded numbers help the model fill arguments correctly and block bad input. A wide-open object schema is a red flag.
Next, error handling. A good server reports tool failures inside the result with an error flag, so the model can recover, instead of crashing the whole call.
Finally, look outside the schema: who publishes the server, can you pin a version, what permissions does it need, and can you see logs in production. Score all of this against a rubric before you say yes.
Vet an MCP server like a production dependency, not a convenience. Score it across five dimensions.
First, schema clarity. The model picks tools from descriptions and argument schemas. Descriptions must be accurate and specific. Schemas should use required fields, enums, and bounded numeric ranges. Wide-open object schemas force the model to guess and invite bad input.
Second, error semantics. A good server signals tool failures inside the result with `isError: true`, not as a protocol crash. That lets the agent read the error and recover gracefully.
Third, auth and least privilege. Tools that touch networks or mutate state must be labeled. The server should request the narrowest scope it needs. Then the host can tier approvals and avoid a confused-deputy problem.
Fourth, supply-chain trust. Check who publishes the server, whether the source is open, whether releases are signed, and whether you can pin a version. An unpinned server can rug-pull a tool definition after you approve it.
Fifth, maintenance and observability. Look for active upkeep, a changelog, and structured logs or tracing. Then turn these into a weighted rubric and require a passing score before any production integration.
Quick reference
TL;DR
Vet an MCP server like a dependency: clear tool schemas the model can pick correctly, honest error semantics, scoped auth, supply-chain trust, and observability, scored against a rubric.
Think of hiring a contractor to work in your house. You don't just take the first one who shows up. You read their job sheet carefully, in case it secretly says 'also rummage through the drawers'. You confirm the work order is specific: exactly what gets fixed, with which tools. You make sure they tell you clearly when a job fails instead of going silent. And you hand them only the one key they need, not the whole keyring. You also check references, and you verify the person who shows up next week is the same one you vetted. An MCP server is a contractor for your AI agent. The same caution applies: read the tool descriptions, demand tight schemas, verify honest error reporting, scope its access, and trust its source before you let it near production.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A throwaway local prototype with a server you authored yourself The supply-chain and signing checks add friction with no real threat; you already control the source and the lifecycle.
A read-only server exposing only public, non-sensitive resources Least-privilege and confused-deputy concerns shrink when there are no secrets and no side effects to abuse.
A vendor server already covered by an org-wide security review Re-running the full rubric duplicates a central gate; defer to the approved registry instead of vetting twice.
Quick scan · 2–4 min
Everything important, quickly.
5 min: frame the server as an untrusted dependency, walk the five rubric dimensions (schema clarity, error semantics, auth and least privilege, supply chain, observability), then map each onto an OWASP MCP risk and a pass-fail gate.
Side-by-side
Criterion
What good looks like
Red flag
Tool description
Accurate, scoped, no hidden directives
Vague or oddly long, embeds instructions
Input schema
Required fields, enums, bounded ranges
Naked object type, no properties
Error semantics
isError true inside the result
Logic failures thrown as protocol errors
Auth scope
Narrowest privilege the tool needs
Broad credentials, shared secrets
Supply chain
Open source, signed, version pinned
Unpinned, unsigned, unknown publisher
Operability
Maintained, logged, traceable
Stale repo, opaque failures
Real-world examples
Real products, models, and research that use this idea.
1The OWASP MCP Top 10, published in 2025, codifies tool poisoning, rug pulls, and confused-deputy risks into a checklist teams use for server vetting.
2Anthropic's MCP server registry lets a host pin a specific server version, so an approved tool definition cannot silently change underneath the agent.
3mcp-inspector lets you enumerate a server's tools and schemas and replay tools/call traffic, so you can audit descriptions and error shapes before integration.
4Cursor and Claude Code surface a per-tool approval UX, which only works when destructive tools are labeled in the server's tool definitions.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you concretely detect tool poisoning in a server's tool descriptions?
Sign in or go Pro to see the approach for this follow-up.
QHow do you defend against a rug pull where a server redefines a tool after you approved it?
Sign in or go Pro to see the approach for this follow-up.
QHow would you turn these criteria into a quantitative adoption rubric a team can apply?
Sign in or go Pro to see the approach for this follow-up.
QWhat observability would you require from an MCP server in production?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Vetting only the schema and skipping supply-chain trust. A clean tool definition from an unpinned, unsigned server can still rug-pull you after approval.