A LoRA adapter passes its full eval harness cleanly but garbles output on roughly one in five production requests after deployment. What is the leading suspect, why does it produce intermittent rather than constant breakage, and how do you prevent the class of bug?
Free · 2 AI evals / day
The leading suspect is a tokenizer mismatch between training and serving.
A fine-tuned adapter is more than just weights. The tokenizer is part of the story: it determines which integer ids map to which strings, especially for special tokens (chat-template markers, EOS, custom additions). When you train, the adapter learns to expect those specific ids in specific positions. When you serve, the inference stack must use the same tokenizer for the adapter to make sense.
The failure mode in production looks intermittent because the drift only matters for requests that touch the affected tokens. Most requests use ordinary text that tokenizes the same way under both old and new tokenizer revisions, so they decode cleanly. But certain requests (those with specific system prompts, tool-call markers, or templated role headers) hit tokens whose ids have shifted, and the model emits garbage on those.
The eval harness misses this entirely because eval loads the trained tokenizer alongside the adapter. The mismatch never appears in eval; it only appears in prod, where the serving stack pulled a different tokenizer revision (often because someone updated the base model from main without realizing the tokenizer had moved with it).
The fix is procedural: ship the tokenizer with the adapter as one artifact, pin the tokenizer to an exact commit hash, and assert at startup that the loaded vocab size and special-token ids match the training snapshot.
**Tokenizer mismatch between training and serving** is the leading suspect.
An adapter is two artifacts: weights and tokenizer. The adapter weights learn to expect specific integer ids for the special tokens that appear in the chat template (turn markers, role headers, tool-call markers, EOS). The eval harness loads both from the training snapshot, so the harness can never see drift. The serving stack often loads the adapter against whatever tokenizer revision lives in its container, which may be a newer pull where special-token ids or chat-template Jinja have shifted.
The failure is intermittent because drift only matters where it bites. Requests whose prompts and outputs avoid the drifted tokens tokenize and decode cleanly; the model handles them as normal. Requests that touch the affected tokens (a specific system prompt template, a tool-call marker the new revision renumbered, a role header that changed) hit ids the adapter was never trained on, and the model emits garbage. One in five maps to the share of templated content paths that use a drifted token, which lines up with typical production traffic mixes.
Prevention is procedural and cheap:
- **Ship the tokenizer with the adapter** as a single versioned artifact, not as a reference to upstream.
- **Pin by commit hash, not branch.** `revision='main'` is a time bomb.
- **Startup assertion.** On model load, check vocab_size, eos_token_id, bos_token_id, and any added special-token ids against the training snapshot. Fail the deploy if they differ.
- **Templated round-trip test as a release gate.** Render a fixed multi-turn conversation through apply_chat_template, compare the token-id sequence against a recorded reference, and refuse to ship if they do not match.
These steps catch the drift before it touches a real user.
Quick reference
TL;DR
Tokenizer mismatch between training and serving. Eval uses the trained tokenizer; prod loaded a drifted version. Requests that touch a drifted special token break; others do not.
Picture two libraries that both speak the same language but use slightly different catalog numbers for some books. You learned to find books by their numbers in one library. Then you try the same numbers at the other library. Most numbers still point to the right book, so most of the time things work fine. But a few numbers now refer to completely different books, and whenever a request happens to need one of those, you grab the wrong thing and the answer comes out scrambled. That is what happens when the model was trained against one tokenizer and the serving system silently loaded a different one: most requests still work, but the ones that touch the drifted tokens come out as nonsense.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
The failure is constant across every production request, not intermittent Constant breakage suggests a weight, serving, or wiring bug rather than tokenizer drift, which is content-dependent.
Eval and prod both use the exact same model artifact path with no upstream pull If artifact loading is identical, tokenizer drift is unlikely; investigate other axes like batching, sampling parameters, or runtime.
The model is being served by an upstream-managed inference endpoint You may not control the tokenizer load; coordinate with the vendor to confirm artifact pinning policies.
Garbled output is correlated with output length rather than prompt content Length-correlated issues point at stop-token learning or generation parameters, not tokenizer mismatch.
Quick scan · 2–4 min
Everything important, quickly.
5 min: name the suspect, explain why the failure is intermittent and content-dependent, contrast eval visibility versus prod blindness, then walk through the four-step prevention recipe.
Side-by-side
Diagnostic signal
Tokenizer drift
Weight bug
Serving framework bug
Eval harness
Always clean
May surface
Usually surfaces
Production breakage rate
Intermittent, content-dependent
Constant or pattern-correlated
Often constant or load-correlated
Repro on a single bad prompt
Yes, reliably
Yes
Often non-deterministic
Fix surface
Tokenizer artifact + pinning
Retrain or revert weights
Patch serving stack
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face Hub revisions can change special-token ids or chat-template Jinja without bumping the model name, which is exactly the upstream drift this bug pattern reflects.
2vLLM and TGI documentation recommends bundling the tokenizer with the served model and pinning the revision; this guidance traces back to repeated production incidents in the LoRA serving community.
3Production fine-tuning vendors like Together and Fireworks treat the tokenizer as a first-class part of the adapter artifact and version it explicitly alongside the weights.
4Open-source projects like Axolotl and Unsloth save the tokenizer with each LoRA checkpoint by default, precisely so serving stacks can load the pair as a unit.
5Practitioners reporting eval-clean-prod-broken on Llama 4 Maverick and Qwen 3.5 fine-tunes in 2026 most commonly trace the bug to a tokenizer revision change after a base-model pull.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a templated round-trip test that catches tokenizer drift before deploy?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the difference between pinning by branch, by tag, and by commit hash on Hugging Face Hub?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the intermittent rate often land near 1 in 5 for this class of bug?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Blaming the model weights, the LoRA rank, or the serving framework when the actual bug lives in the tokenizer. The eval harness cannot see the drift because it uses the trained tokenizer.