Fill in the blanks: the special tokens that control sequence boundaries and conversation structure in modern LLMs.

Fill in blank·Medium·4.0 · 0·~1 min·Relevant atMeta OpenAI

Attempt it

If a chat model generates output indefinitely without stopping, the most likely cause is a missing token. To align sequences of different lengths into a batch, models use tokens that must be from the attention computation. In instruction-tuned models, the role boundaries between system, user, and assistant turns are marked by tokens specific to each model's chat template.

Quick reference

TL;DR

EOS stops generation, PAD aligns batches but must be masked out of attention, and per-model role markers fence system, user, and assistant turns in the chat template.

Memory aid

Easy to grasp

Picture writing letters that all have to fit in identically sized envelopes. The 'sincerely, the end' line is what tells the reader to stop; without it they would keep reading whatever scribbles came next. To make short letters fill the same envelope you stuff in blank paper, and you tell the reader to skip the blank pages. And on a group letter, you label each paragraph with who wrote it, so nobody confuses the boss's note with a coworker's. In a language model those three jobs belong to the end token, the padding token, and the role-marker tokens. Each model brands its own labels, so you cannot mix one model's tags into another.

Key concepts

Special tokens Chat templates

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

2-3 min: EOS as the learned stop + PAD batch alignment + attention masking math + per-model role markers + apply_chat_template as the safe inserter.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Transformers raises generation issues when a model's generation_config lacks the right eos_token_id, producing output that runs to max_new_tokens.
vLLM batches requests of different lengths and relies on correct PAD masking so one request never attends into another's padded slots.
OpenAI's ChatML format used by GPT-5.5 fine-tunes fences each turn with <|im_start|> and <|im_end|> role markers from the model's template.
Llama 4 Maverick ships its own chat template, so apply_chat_template inserts its specific role markers rather than the ChatML strings.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you cannot change the model, how do you stop runaway generation without a reliable EOS?

QHow does left padding versus right padding interact with the attention mask during generation?

QWhat goes wrong if you reuse one model's role markers when serving a different model family?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing PAD is harmless filler. Unmasked PAD positions still receive softmax weight, so they leak noise into the value-weighted sum and shift outputs.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.