Your input is 9,000 tokens but max_length is 8,192. What are your truncation options?

Flashcard·Easy·4.0 · 0·~30s·Asked atPhonepe Qdrant Tata Digital·Relevant atOpenAI

Attempt it

Truncation drops tokens from an input that exceeds max_length so the resulting sequence fits the model's context window. HuggingFace exposes four named strategies: longest_first (default) trims from the longer of paired inputs until both fit, only_first always trims the first input, only_second always trims the second, and do_not_truncate does not cut at all, so an input over the limit fails later at the model. In chat applications, a manual middle truncation strategy is common: keep the system prompt and the most recent turns, and drop the middle turns. The recurring production bug is silent truncation: the tokenizer cuts content without warning the application, and answers degrade without an obvious signal. Always log when truncation fires.

Your answer

**Truncation** is what the tokenizer does when an input has more tokens than the model can fit. Every model has a maximum context length (`model_max_length`). An input over that limit must be cut or rejected. HuggingFace offers four named strategies, set with the `truncation` argument: - `longest_first` (the default when you write `truncation=True`) trims from the longer of a paired input one token at a time until both fit. - `only_first` always trims the first input in a pair. - `only_second` always trims the second input. - `do_not_truncate` does not cut; the model fails later on the long input. For a single input (not a pair), the three cutting strategies all drop tokens from the end. In chat applications, a typical pattern is **manual middle truncation**: keep the system prompt at the start, keep the most recent few turns at the end, and drop turns from the middle. The middle turns are usually the least important to the current reply. The recurring production bug is silent truncation. The tokenizer cuts content without warning the application. Answers get worse in a way that is hard to trace. Always log when truncation fires by comparing token counts before and after.

**Truncation** drops tokens from an input that exceeds `max_length` so the result fits the model's context window. The limit is `model_max_length` or the application's own budget. HuggingFace exposes four named strategies on the `truncation` argument. `longest_first`, the default for `truncation=True`, trims from the longer of paired inputs one token at a time. It suits question and context pairs in retrieval where you want both sides present. `only_first` always cuts from the first input; `only_second` always cuts from the second; both are useful when one side carries the essential content. `do_not_truncate` does not cut, so the model fails later; add your own length check when silent loss is unacceptable. For a single input the cutting strategies all trim from the end. The truncation side defaults to `right` (drop the tail). But `tokenizer.truncation_side = 'left'` flips it for use cases where the tail matters most, like chat history where the most recent turns are at the end. None of the built-in strategies handle chat history well. The standard production pattern is manual middle truncation. Keep the system prompt at the top and the recent few user and assistant turns. Drop turns from the middle by oldest first. The system prompt and the recent context are usually what the model needs. The recurring production bug is silent truncation. The tokenizer cuts and returns ids; the application has no idea content was dropped, and quality drops without an obvious symptom. Always log when truncation fires by comparing token counts before and after truncation.

TL;DR

Truncation drops tokens from an over-long input so it fits max_length; HF strategies are longest_first, only_first, only_second, do_not_truncate. Silent truncation is the recurring production bug.

Memory aid

Easy to grasp

Imagine a backpack that fits ten books. Someone hands you fifteen. You have to leave some behind, but you cannot leave any specific book at random. A smart traveler thinks about which books matter: the map at the top stays, the trip notes at the bottom stay, and the duplicates in the middle get left at the hotel. Truncation in tokenization is the same calculation. The backpack is the model's context window. The books are tokens. The strategy is the rule for which tokens to leave behind. And if you never tell the traveler the bag was overstuffed, they leave things behind silently and you only notice at the destination.

Key concepts

Truncation Context window Max length

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Truncation is the tokenizer's answer to a hard constraint. The model has a maximum context length, and any input over that limit has to either be cut or rejected. The key choices are which side to cut from, which API surface implements the cut, and how to make sure the cut does not happen silently.

What truncation is and what forces it

Every model has a maximum context length. For Llama 3 it is 8k for the base 8B model and 128k for the long-context fine-tunes. GPT-5.5 and Claude Opus 4.7 offer 1M tokens through the API, and Gemini 3.1 Pro offers about 1M. These are hard architectural and serving limits, not soft suggestions: an input that exceeds the limit cannot be processed at all.

The tokenizer is the layer that enforces this. The application calls tokenizer(text, truncation=True, max_length=N) and the tokenizer guarantees the returned input_ids will be at most N. To honor that guarantee, it drops tokens above N.

Truncation also applies to budgets smaller than the model's full limit. Production systems usually keep some headroom for the response. Even a 128k model might be configured to truncate inputs at 100k to leave 28k for generation. The same machinery applies.

The four named HuggingFace strategies

Manual middle truncation for chat

Silent truncation is the production bug class

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

HuggingFace tokenizer(text, truncation=True, max_length=8192) uses longest_first by default and returns overflowing_tokens if you ask for them.
LangChain and LlamaIndex chat memory implementations apply manual middle truncation against a token budget computed from the model's context window.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is longest_first the default and when is it the wrong choice?

It was designed for question plus context pairs in pretraining era encoder workloads. There, both sides are roughly equally important and you want them both present. It is wrong when one side is structurally precious (the question) and the other side is the place to absorb cuts (the context).

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Calling tokenizer(text, truncation=True) and never logging when truncation fires. The tokenizer silently drops tokens past max_length and answers degrade with no obvious signal.

60 second bullets to scan on the way to the call.

Define truncation and why max_length makes it necessary.
Name the four HuggingFace truncation strategies and what each one does.

Primary sources. Browse if you want the original framing.