Your input is 9,000 tokens but max_length is 8,192. What are your truncation options? · Zenaique

Attempt it

Truncation drops tokens from an input that exceeds max_length so the resulting sequence fits the model's context window. HuggingFace exposes four named strategies: longest_first (default) trims from the longer of paired inputs until both fit, only_first always trims the first input, only_second always trims the second, and do_not_truncate raises an error instead of cutting. In chat applications, a manual middle truncation strategy is common: keep the system prompt and the most recent turns, drop the middle turns. The recurring production bug is silent truncation: the tokenizer cuts content without warning the application, and answers degrade without an obvious signal. Always log when truncation fires.

Your answer

**Truncation** is what the tokenizer does when an input has more tokens than the model can fit. Every model has a maximum context length (`model_max_length`), and any input over that limit either has to be cut down or rejected. HuggingFace exposes four named truncation strategies, picked via the `truncation` argument: - `longest_first` (the default when you write `truncation=True`) trims from the longer of a paired input one token at a time until both fit. - `only_first` always trims the first input in a pair. - `only_second` always trims the second input. - `do_not_truncate` raises an error if the input is over `max_length` instead of cutting. For a single input (not a pair), the strategies all collapse to the same behavior: drop tokens from the end. In chat applications, the named strategies are rarely enough by themselves. A typical pattern is **manual middle truncation**: keep the system prompt at the start, keep the most recent few turns at the end, and drop turns from the middle. The middle turns are usually the least important to the current reply. The recurring production bug is silent truncation: the tokenizer cuts content without warning the application, the model never sees the dropped tokens, and answers degrade in a way that is hard to trace. Always log when truncation fires, by checking the difference between the pre truncation and post truncation token counts.

**Truncation** drops tokens from an input that exceeds `max_length` so the result fits the model's context window. It is the tokenizer's response to the hard limit imposed by `model_max_length` or by the application's chosen budget. HuggingFace exposes four named strategies on the `truncation` argument. `longest_first`, the default for `truncation=True`, trims from the longer of paired inputs one token at a time; useful for question and context pairs in retrieval where you want both sides to be present. `only_first` always cuts from the first input; `only_second` always cuts from the second; both are useful when one side carries the irreducible content. `do_not_truncate` raises an error rather than cutting, which is the right choice when silent loss is unacceptable. For a single input these all reduce to trimming from the end. The truncation side defaults to `right` (drop the tail) but `tokenizer.truncation_side = 'left'` flips it for use cases where the tail matters most, like chat history where the most recent turns are at the end. None of the built in strategies handle chat history well. The standard production pattern is manual middle truncation: keep the system prompt at the top, keep the recent few user and assistant turns, drop turns from the middle by oldest first. The system prompt and the recent context are usually what the model needs. The recurring production bug is silent truncation. The tokenizer cuts and returns ids; the application has no idea content was dropped, and quality degrades without an obvious symptom. Always log when truncation fires by comparing pre truncation and post truncation token counts, or by checking the `overflow_to_sample_mapping` and `num_truncated_tokens` fields the tokenizer can return.

Quick reference

TL;DR

Truncation drops tokens from an over-long input so it fits `max_length`; HF strategies are `longest_first`, `only_first`, `only_second`, `do_not_truncate`. Silent truncation is the recurring production bug.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a backpack that fits ten books. Someone hands you fifteen. You have to leave some behind, but you cannot leave any specific book at random. A smart traveler thinks about which books matter: the map at the top stays, the trip notes at the bottom stay, and the duplicates in the middle get left at the hotel. Truncation in tokenization is the same calculation. The backpack is the model's context window. The books are tokens. The strategy is the rule for which tokens to leave behind. And if you never tell the traveler the bag was overstuffed, they leave things behind silently and you only notice at the destination.

Key concepts

Truncation Context window Max length

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define truncation, name the four HuggingFace strategies, explain the side argument, describe manual middle truncation for chat, and close on silent truncation as the production bug class.

Real-world examples

Real products, models, and research that use this idea.

HuggingFace `tokenizer(text, truncation=True, max_length=8192)` uses `longest_first` by default and returns `overflowing_tokens` if you ask for them.
LangChain and LlamaIndex chat memory implementations apply manual middle truncation against a token budget computed from the model's `context_length`.
OpenAI and Anthropic chat APIs reject requests that exceed the model's context window with a 400 level error rather than silently truncating.
vLLM logs `truncation` events at the engine level when an input over `max_model_len` is submitted with truncation enabled.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is `longest_first` the default and when is it the wrong choice?

Sign in or go Pro to see the approach for this follow-up.

QHow would you implement middle truncation for a 200 turn chat history against a 32k token budget?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Calling `tokenizer(text, truncation=True)` and never logging when truncation fires. The tokenizer silently drops tokens past `max_length` and answers degrade with no obvious signal.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.