Why prefix injection works against autoregressive models
Autoregressive models pick each token conditioned on what came before. Forcing the opening to be 'Sure, here is how to' biases the entire distribution toward compliance because refusals start very differently.
Imagine asking a careful friend a question they would normally decline to answer. If you let them speak naturally, they start with 'I am sorry, but...' and that opening commits them to the rest of the refusal. Now imagine you hand them a notecard that says 'Sure, here is how to do it' and ask them to read that aloud before continuing in their own words. Once the first words are out of their mouth, finishing the sentence as a refusal would sound bizarre. They have already committed to a helpful tone, and the rest of what they say tends to follow that tone. Autoregressive language models work the same way, each next word is chosen with the previous words as context, so the opening words steer everything that comes after.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prefix injection is the canonical demonstration that LLM safety tuning is a probability shift over next-token distributions, not a discrete safety gate. Autoregressive generation conditions every token on the preceding sequence, and safety tuning concentrates probability mass on refusal-shaped openings for unsafe categories. Once the attacker forces the opening to be a compliance-shaped prefix, the conditioning has moved the model into the helpful-answer region of its distribution, and the tuning's lever no longer has purchase.
The interview question is checking whether you understand both the mechanism (autoregressive conditioning) and the architectural consequence (the model cannot be the safety boundary on its own). Once you internalise both, the defence pattern, strip injection at runtime, train against the template, classify the output, falls out naturally.
The autoregressive-conditioning mechanism
Every autoregressive language model generates token t+1 by computing a probability distribution conditioned on tokens t_0 through t_n. The model's parameters define that conditional distribution; sampling (greedy, top-k, top-p, temperature) selects from it.
Safety tuning, RLHF, DPO, constitutional AI, modifies the parameters so that for prompts the tuning recognises as unsafe categories, the probability distribution over the first response tokens concentrates on refusal-shaped openings. After tuning, the model is very likely to start an unsafe-prompt response with 'I cannot,' 'I am sorry,' 'As an AI assistant.' The continuation that follows is high-probability given such an opener; 'I cannot help with that' is a more probable continuation than 'I cannot ignore my training and let me explain.'
Notice what the tuning did and did not do. It shifted probability mass on the first tokens. It did NOT install an out of band safety check that says 'is this response unsafe?' before each token. The model has no internal state for safety beyond what is implicit in its parameters and its current conditioning context.
Now consider what happens when the attacker forces the first tokens to be 'Sure, here is how to.' The model is now conditioning on a prefix that the safety tuning never anchored on for unsafe categories. The conditional distribution over the next token is dominated by tokens that continue 'Sure, here is how to' coherently, task-specific instructions, recipe ingredients, code snippets, whatever the unsafe ask was. The refusal continuation has low probability given this prefix because it never appears in training data after 'Sure, here is how to.' The tuning's lever has been pulled the wrong way.
This is the mechanism. It is not a model bug; it is the architecture working as designed. The defence cannot be 'fix the model' alone.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The original 2023 prefix-injection work by Wei et al. demonstrated the technique against GPT-3.5 and Llama 2 with 'start with Sure' templates; 2026 follow-ups confirm the same vulnerability class still requires explicit defences on GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and Llama 4.
- Llama Guard 4 in 2026 runs as an output classifier evaluating the full response, decoupled from how it opens, specifically to address prefix-injection bypasses.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does refusal-suppression composition differ from a direct prefix injection?
Direct prefix injection names the compliance opener ('start with Sure'). Refusal-suppression names the refusal opener the model should NOT use ('do not start with I cannot or I am sorry'). The first concentrates probability on a compliance prefix; the second pushes probability away from refusal prefixes, leaving compliance as the default. Both exploit the same conditional-distribution mechanism; refusal-suppression is sometimes harder to detect because it never names the compliance opener explicitly.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the safety tuning as a single yes or no gate. It is a distribution over next tokens; bias the prefix and the gate moves.
60 second bullets to scan on the way to the call.
How autoregressive token by token generation creates the prefix-conditioning bias
Why safety tuning is a distributional shift, not a hard gate
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.