Tool-calling fine-tune: data shape and the common pitfall

Attempt it

You're fine-tuning a 7B model to do tool-calling reliably for a specific set of tools (search, calculator, db_query). What's the shape of the training data, and what's the most common pitfall teams hit?

Free · 2 AI evals / day

Quick reference

TL;DR

Train on system-message tool schemas plus special-token wrapped call JSON, supervise multi-turn result flows, and mix in 30-50% no-tool examples so the model learns when NOT to call.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine teaching an intern to use office tools: a phone, a calculator, a filing cabinet. You show them request and action pairs so they learn the exact form each action takes. If every example you show involves grabbing a tool, the intern concludes that every question needs a tool. Then someone asks 'what is two plus two' and they reach for the calculator anyway. The fix is to also show plenty of moments where the right move is to just answer from their own head. The hard skill is not the form of the action. It is the judgment call about whether any tool is needed at all.

Key concepts

Tool / function calling No-tool negative examples Chat template and special tokens Constrained decoding

Quick scan · 2–4 min

Everything important, quickly.

5 min: data triple structure + special-token call format + multi-turn result turns + the positive-only pitfall + the 30 to 50 percent no-tool mix + secondary token and format traps + constrained-decoding pairing.

Code example

json

{
  "messages": [
    {"role": "system", "content": "Tools: search(q), calculator(expr), db_query(sql)"},
    {"role": "user", "content": "What's 2+2?"},
    {"role": "assistant", "content": "4"}
  ],
  "note": "no-tool negative example: answer directly, emit NO tool call"
}

Side-by-side

Aspect	The easy part (format)	The hard part (the gate)
What it is	Special-token-wrapped JSON call shape	Deciding whether any tool is needed
How you teach it	Positive call examples in the chat template	30 to 50 percent no-tool negatives
Failure if ignored	Calls fail to parse or never fire	Tools fire on trivial queries, cost explodes
Backstop at inference	Constrained decoding to the schema	Routing or confidence threshold on the call

Real-world examples

Real products, models, and research that use this idea.

NousResearch Hermes-format tool calling wraps the call JSON in <tool_call> tags, the convention many open fine-tunes copy onto Llama 4 and Qwen bases.
Berkeley's Gorilla and the Berkeley Function-Calling Leaderboard benchmark exactly this skill, including 'irrelevance' cases where the right answer is no tool.
OpenAI's fine-tuning API supports function-calling examples on GPT-5.5-mini so teams can bias both the call format and the when to call decision.
vLLM and TGI ship guided-decoding backends (Outlines, XGrammar) that constrain tool-call output to a JSON schema, the standard pairing with a tool-calling fine-tune.
Anthropic's tool-use API on Claude Opus 4.7 documents a strict tool_result content-block shape that any local fine-tune must mirror to stay runtime-compatible.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you measure whether the no-tool gate actually improved after fine-tuning?

Sign in or go Pro to see the approach for this follow-up.

QWhy pair the fine-tune with constrained decoding instead of trusting the model to emit valid JSON?

Sign in or go Pro to see the approach for this follow-up.

QHow would you supervise a multi-call flow where one tool's output feeds the next call?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes if the inference runtime uses a different chat template than your training data?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.