The correct answer is **`add_special_tokens=False`**, and the trick is that the other three options are invented parameter names that merely sound official.
Here is the mechanism. A HuggingFace tokenizer can recognize certain strings, like `<|system|>` or `[INST]`, as special tokens and turn them into dedicated token ids. By default the flag `add_special_tokens` is True, which means the tokenizer is allowed to make those conversions and will also add things like a beginning of sequence token.
When the input is text a user typed, you do not want any of that. If a user writes `<|system|>` and the tokenizer turns it into the genuine system id, the model now sees a system instruction that the user smuggled in. Passing `add_special_tokens=False` keeps those strings as ordinary characters, so they cannot pose as structural tokens.
The wrong options, `sanitize_special_tokens`, `user_mode`, and `escape_html`, are distractors. They are not parameters the tokenizer actually exposes, which is why memorizing real API signatures pays off here.
The answer is **`add_special_tokens=False`**.
HuggingFace tokenizers take an `add_special_tokens` boolean on `encode()` and `__call__()`. When it is True, the default, the tokenizer maps recognized strings such as `<|system|>` or `[INST]` to their special token ids and also injects sequence-boundary tokens. When it is False, those strings stay as plain characters and only ordinary subword tokenization applies.
The security relevance is direct. User text should never be allowed to mint structural tokens, because those tokens define role boundaries the model trusts. Tokenizing user content with `add_special_tokens=False` closes the Special Token Injection path, while you reserve `add_special_tokens=True` for the scaffolding you control.
The other three options are not real tokenizer parameters. The sanitize, user-mode, and HTML-escape names are distractors built to sound like security features. HTML-escaping in particular would not help, since the danger is at the token-id layer, not in browser rendering. Knowing the actual API signature is what separates the right answer from the plausible decoys.
Quick reference
TL;DR
Tokenize user content with add_special_tokens=False so strings like <|system|> stay literal text instead of becoming the real special token ids that break the trust boundary.
Imagine a theater where saying the phrase 'lights out' really kills the stage lights, because the crew is trained to obey it. Now a heckler in the audience shouts 'lights out' and the room goes dark, because the crew could not tell the heckler apart from the director. The fix is a rule: anything an audience member says is just words, never a command. In a language model, the dangerous command-words are special tokens like the system marker. Setting add_special_tokens to False is that rule, so a marker typed by a user is treated as plain text rather than a real instruction the model obeys.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tokenizing the structural template you author yourself Your own scaffolding needs the real BOS, EOS, and role markers, so forcing add_special_tokens=False there strips the boundaries you intend.
Relying on the flag as the only defense in a multi-step prompt builder A later join or re-tokenize under the default can re-mint markers, so the flag must be paired with structural separation.
Web rendering concerns where escaping is genuinely needed escape_html guards the browser, not the tokenizer; it is the right tool for a different problem, not for token-layer injection.
Models whose tokenizer does not register the marker strings as special If the strings were never special ids, the flag changes nothing, and you need vocabulary-aware validation instead.
Quick scan · 2–4 min
Everything important, quickly.
2-3 min: what add_special_tokens does + default True is unsafe + why escape_html misses the layer + apply_chat_template pairing + structural separation as defense in depth.
Real-world examples
Real products, models, and research that use this idea.
1HuggingFace Transformers exposes add_special_tokens on tokenizer.encode and __call__, and the docs warn against minting special ids from untrusted input.
2LangChain and similar orchestration layers that build prompts from user fields can leak markers if the underlying tokenize step keeps the default add_special_tokens=True.
3OpenAI's messages API for GPT-5.5 keeps each user turn structurally separate so user text cannot mint a system boundary on its own.
4Anthropic's Claude Opus 4.7 messages format likewise isolates user content, which is the API-level version of the add_special_tokens=False defense.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QEven with add_special_tokens=False, how could a developer still reintroduce the injection risk?
Sign in or go Pro to see the approach for this follow-up.
QWhy is structural separation at the messages API stronger than relying on a single flag?
Sign in or go Pro to see the approach for this follow-up.
QHow would you detect attempted special-token injection in production logs?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Picking a plausible-sounding flag like sanitize_special_tokens or escape_html. Those are not real tokenizer parameters; the real knob is add_special_tokens.