Fill in the blanks: the special tokens that control sequence boundaries and conversation structure in modern LLMs. · Zenaique

Attempt it

If a chat model generates output indefinitely without stopping, the most likely cause is a missing token. To align sequences of different lengths into a batch, models use tokens that must be from the attention computation. In instruction-tuned models, the role boundaries between system, user, and assistant turns are marked by tokens specific to each model's chat template.

Quick reference

TL;DR

EOS stops generation, PAD aligns batches but must be masked out of attention, and per-model role markers fence system, user, and assistant turns in the chat template.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture writing letters that all have to fit in identically sized envelopes. The 'sincerely, the end' line is what tells the reader to stop; without it they would keep reading whatever scribbles came next. To make short letters fill the same envelope you stuff in blank paper, and you tell the reader to skip the blank pages. And on a group letter, you label each paragraph with who wrote it, so nobody confuses the boss's note with a coworker's. In a language model those three jobs belong to the end token, the padding token, and the role-marker tokens. Each model brands its own labels, so you cannot mix one model's tags into another.

Key concepts

Special tokens Chat templates

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

2-3 min: EOS as the learned stop + PAD batch alignment + attention masking math + per-model role markers + apply_chat_template as the safe inserter.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Transformers raises generation issues when a model's generation_config lacks the right eos_token_id, producing output that runs to max_new_tokens.
vLLM batches requests of different lengths and relies on correct PAD masking so one request never attends into another's padded slots.
OpenAI's ChatML format used by GPT-5.5 fine-tunes fences each turn with <|im_start|> and <|im_end|> role markers from the model's template.
Llama 4 Maverick ships its own chat template, so apply_chat_template inserts its specific role markers rather than the ChatML strings.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you cannot change the model, how do you stop runaway generation without a reliable EOS?

Sign in or go Pro to see the approach for this follow-up.

QHow does left padding versus right padding interact with the attention mask during generation?

Sign in or go Pro to see the approach for this follow-up.

QWhat goes wrong if you reuse one model's role markers when serving a different model family?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing PAD is harmless filler. Unmasked PAD positions still receive softmax weight, so they leak noise into the value-weighted sum and shift outputs.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.