Describe how a Special Token Injection (STI) attack works and what application-layer mitigations can be applied.

Attempt it

Explain the mechanism of a Special Token Injection (STI) attack: how does unsanitized user input become a security vulnerability at the tokenizer level? Describe at least two concrete mitigations that can be applied before or during tokenization.

Free · 2 AI evals / day

Quick reference

TL;DR

STI exploits the gap between a special token's string and its id: tokenized under add_special_tokens=True, user-typed markers become real structural ids the model trusts as legitimate boundaries.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of a building where doors open only for staff badges. An attacker prints a fake badge that looks exactly like the real one, and the scanner cannot tell the difference, so the fake opens secure doors. In a language model, the special tokens like the system marker are the badges that grant authority over the conversation. If user text containing a marker is scanned the normal way, the model accepts that fake badge as real and follows it. The defenses are to refuse to read user badges at all, to tear up any badge-looking text a user submits, and to keep visitors in a separate lane where they could never carry a staff badge.

Key concepts

Special tokens LLM security Prompt injection

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3-4 min: string vs id gap + forged-boundary consequence + add_special_tokens=False + input sanitization + structural separation + the re-tokenization pitfall.

Real-world examples

Real products, models, and research that use this idea.

HuggingFace Transformers exposes add_special_tokens on encode and __call__, which is the primary knob teams flip to False for untrusted input.
OWASP's Top 10 for LLM Applications catalogs prompt injection, of which special-token injection is the tokenizer-layer instance.
OpenAI's GPT-5.5 messages API keeps each user turn structurally separate so user text cannot mint a system boundary by itself.
Anthropic's Claude Opus 4.7 messages format isolates user content the same way, which is structural separation enforced at the API layer.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build a tokenizer wrapper that provably blocks STI across every code path?

Sign in or go Pro to see the approach for this follow-up.

QCan STI succeed against a model served only through a strict messages API with no manual string assembly?

Sign in or go Pro to see the approach for this follow-up.

QHow do you monitor for attempted STI without flooding logs with false positives?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing STI with ordinary prompt injection. STI is the tokenizer-layer variant where user strings become genuine structural ids, not just persuasive natural-language text.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.