EOS stops generation, PAD aligns batches but must be masked out of attention, and per-model role markers fence system, user, and assistant turns in the chat template.
Picture writing letters that all have to fit in identically sized envelopes. The 'sincerely, the end' line is what tells the reader to stop; without it they would keep reading whatever scribbles came next. To make short letters fill the same envelope you stuff in blank paper, and you tell the reader to skip the blank pages. And on a group letter, you label each paragraph with who wrote it, so nobody confuses the boss's note with a coworker's. In a language model those three jobs belong to the end token, the padding token, and the role-marker tokens. Each model brands its own labels, so you cannot mix one model's tags into another.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This fill in the blank looks like vocabulary recall, but it is really a diagnostic exercise. Each blank is anchored to a symptom: output that never stops, batches that must be squared off, and turns that must be fenced. If you reason from the symptom to the token, you never have to memorize a list.
The deeper point is that these tokens live in the tokenizer and serving layers, not in the prompt text. The model never sees your nicely formatted string. It sees an id sequence that the tokenizer assembled, and the special tokens are the scaffolding it added. Understanding that boundary is what turns this from trivia into a debugging skill.
There is a reason interviewers like this exact framing. Each blank corresponds to a real on-call story: a chatbot that would not stop talking, a batched endpoint whose answers got worse under load, and a model swap that broke role-following. Connecting the token to the incident is what separates someone who memorized a glossary from someone who has actually shipped and debugged a serving stack.
Blank one: EOS and the runaway loop
The clue in the prompt is 'generates output indefinitely without stopping'. That is the signature of a missing or wrong end token.
Autoregressive decoding samples one token, appends it, and repeats. The only thing that ends this loop from inside the model is sampling EOS. The model was trained to make EOS likely once a response has reached a natural end, so a healthy model stops on its own.
When EOS is missing from the generation config, or the model was never taught to emit it, the loop only ends at the max_new_tokens ceiling. The output looks truncated or repetitive, and people often respond by raising the cap, which makes the bill worse without fixing anything. The real fix is restoring the correct EOS id and confirming the model emits it.
A subtle variant of this bug comes from models with more than one end token. A chat model might define both a generic end of text id and a turn-level id such as <|eot_id|>. If your decoding loop only watches for one, the model can emit the other and keep going. This is why production frameworks accept a list of stop ids, and why blindly reusing one model's generation config on another family is a frequent cause of the runaway symptom this blank describes.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face Transformers raises generation issues when a model's generation_config lacks the right eos_token_id, producing output that runs to max_new_tokens.
- vLLM batches requests of different lengths and relies on correct PAD masking so one request never attends into another's padded slots.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you cannot change the model, how do you stop runaway generation without a reliable EOS?
Think about stopping criteria the decoder exposes beyond the learned token, and their tradeoffs.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing PAD is harmless filler. Unmasked PAD positions still receive softmax weight, so they leak noise into the value-weighted sum and shift outputs.
60 second bullets to scan on the way to the call.
Why runaway generation points to a missing EOS token
The job PAD tokens do when batching variable-length prompts
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.