What criteria would you use to evaluate the quality of an MCP server's tool definitions before integration?
You are evaluating an MCP server before integrating it into a production agent. What criteria would you use to assess the quality of its tool definitions? Name at least five.
Vet an MCP server like a dependency: clear tool schemas the model can pick correctly, honest error semantics, scoped auth, supply-chain trust, and observability, scored against a rubric.
Think of hiring a contractor to work in your house. You don't just take the first one who shows up. You read their job sheet carefully, in case it secretly says 'also rummage through the drawers'. You confirm the work order is specific: exactly what gets fixed, with which tools. You make sure they tell you clearly when a job fails instead of going silent. And you hand them only the one key they need, not the whole keyring. You also check references, and you verify the person who shows up next week is the same one you vetted. An MCP server is a contractor for your AI agent. The same caution applies: read the tool descriptions, demand tight schemas, verify honest error reporting, scope its access, and trust its source before you let it near production.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
MCP makes integration cheap, and that is exactly why it is dangerous in production. A single config line connects your agent to a third-party process that can advertise tools, hand the model instructions, and act with whatever credentials you grant it. The model reads tool descriptions and schemas at face value, so a server's tool definitions are not just documentation. They are an attack surface and a correctness surface at the same time.
The right mental model is that an MCP server is an untrusted dependency, closer to a browser extension than to a library you compiled yourself. You would not ship an unreviewed extension to every user. You should not wire an unvetted server into a production agent either.
This deep dive builds an adoption rubric across five dimensions: schema clarity so the model picks correctly, error semantics so the agent recovers, auth and least privilege so a compromise stays contained, supply-chain trust so an approved server cannot mutate under you, and observability so you can see what actually happened. Each dimension maps onto a concrete OWASP MCP Top 10 risk.
Why tool definitions are a correctness and security surface
When a host connects to an MCP server, it calls tools/list and registers each entry as a function-callable tool. The model then chooses among those tools using only the description and the input schema. There is no out of band channel. Whatever the server writes into a description, the model treats as ground truth about what the tool does.
That creates two distinct failure modes. The first is a correctness failure: a vague or overlapping description makes the model pick the wrong tool or fill arguments badly. The second is a security failure: a malicious description can embed instructions aimed at the model rather than the human reading the catalog. This is tool poisoning, and it is the single most MCP-specific risk on the list.
So the first rubric dimension is description integrity. Descriptions must be accurate, scoped to one job, and free of imperative text directed at the model. A suspiciously long description, encoded blobs, or text that references other tools and files is a hard stop. You audit this by enumerating the catalog with a tool like mcp-inspector and reading every description as if it were code review.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Criterion | What good looks like | Red flag |
|---|---|---|
| Tool description | Accurate, scoped, no hidden directives | Vague or oddly long, embeds instructions |
| Input schema | Required fields, enums, bounded ranges | Naked object type, no properties |
| Error semantics | isError true inside the result | Logic failures thrown as protocol errors |
| Auth scope | Narrowest privilege the tool needs | Broad credentials, shared secrets |
| Supply chain | Open source, signed, version pinned | Unpinned, unsigned, unknown publisher |
| Operability | Maintained, logged, traceable | Stale repo, opaque failures |
Real products, models, and research that use this idea.
- The OWASP MCP Top 10, published in 2025, codifies tool poisoning, rug pulls, and confused-deputy risks into a checklist teams use for server vetting.
- Anthropic's MCP server registry lets a host pin a specific server version, so an approved tool definition cannot silently change underneath the agent.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you concretely detect tool poisoning in a server's tool descriptions?
Diff descriptions against expected behavior; flag imperative text aimed at the model, encoded payloads, or instructions referencing other tools or files.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Vetting only the schema and skipping supply-chain trust. A clean tool definition from an unpinned, unsigned server can still rug-pull you after approval.
60 second bullets to scan on the way to the call.
Why tool descriptions are a correctness surface for model tool selection
What makes an input schema tight versus dangerously permissive
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.