Describe what offset_mapping returns from a HuggingFace fast tokenizer. Give two concrete use cases where offset_mapping is required, and explain what problem arises if you try to implement the same functionality without it using a slow tokenizer.
Free · 2 AI evals / day
A tokenizer chops your text into tokens and gives back integer IDs, but it throws away one useful thing along the way: where each token sat in the original string. The `offset_mapping` feature puts that information back.
When you call a HuggingFace fast tokenizer with `return_offsets_mapping=True`, you get one `(start, end)` pair per token. These are character positions into the input string. Tokenizing `Hello world` gives tokens like `Hello` and ` world` with offsets `(0, 5)` and `(5, 11)`. So token 1 covers characters 0 through 5 of the original text.
Why do you care? Because models predict things at the token level, but humans read at the character level. A named-entity model says token 3 is a person name. To draw a highlight in the UI, you need to know which characters token 3 covers. That is exactly what `offset_mapping` tells you.
The word fast matters here. Fast tokenizers are written in Rust and track these offsets as they tokenize. The older slow tokenizers do not, so you would have to reconstruct positions yourself, which is fiddly and easy to get wrong.
**offset_mapping** is a per-token list of `(start, end)` character indices into the original input string, returned by HuggingFace fast tokenizers when you pass `return_offsets_mapping=True`. Token i covers `text[start:end]`, so it bridges the gap between what the model predicts (token positions) and what the user sees (characters).
Two use cases make it essential. In token classification like NER, the model tags each token, and you map the tagged indices back to character spans to highlight or extract the entity. In extractive QA, the model predicts a start token and an end token, and you recover the answer as `text[offsets[start][0]:offsets[end][1]]`.
Without it, on a slow tokenizer, you reconstruct positions by hand. That breaks on WordPiece `##` continuation pieces. It breaks on byte-level BPE where a token is a fragment of a multi-byte character. And it breaks on any Unicode where character index does not equal byte index. Fast tokenizers compute the mapping in their Rust core, so it is exact and free. The slow path is the kind of code that passes your English unit tests and then mangles spans on accented or CJK input in production.
Quick reference
TL;DR
offset_mapping returns per-token (start, end) character spans, letting you map model predictions back onto the original text for NER, extractive QA, and highlighting.
Imagine highlighting a sentence in a book with sticky tabs, where each tab covers a few letters. Later someone tells you 'the answer is on tab 3 through tab 5', but you need to point at the actual words on the page. You need a little map that says 'tab 3 starts at letter 12 and ends at letter 17'. That map is offset_mapping. The fast tokenizer hands it to you for free; without it you would have to count letters by hand and you would miscount whenever spaces, accents, or emoji get involved.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Free-text generation or summarization where output is never aligned back to input characters If no prediction has to round-trip to a source span, the offsets are unused overhead; passing return_offsets_mapping=True just allocates a list you throw away.
A model whose checkpoint ships only a slow Python tokenizer Slow tokenizers cannot produce offset_mapping at all, so the feature is simply unavailable; you must convert to a fast tokenizer or compute alignment another way.
Aligning to a normalized or transformed string rather than the raw input Offsets index the original pre-normalization text, so if your downstream code works on lowercased or accent-stripped text the spans will not line up; track the transform separately.
Whole-sequence classification like sentiment where there is one label for the whole input There is no per-token span to recover, so token to character mapping adds nothing; you only need the pooled representation, not offsets.
Word-level alignment across two tokenizations rather than token to character For aligning model wordpieces to upstream word indices you want word_ids(), not raw character offsets, which would force you to re-derive word boundaries by hand.
Quick scan · 2–4 min
Everything important, quickly.
3 min: what offsets are + NER mapping + QA span extraction + why decoded-token reconstruction fails + the Unicode/byte trap + special-token (0,0) handling.
Code example
python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-cased") # fast by default
text = "Ada Lovelace wrote the first algorithm."
enc = tok(text, return_offsets_mapping=True)
# Suppose the NER model tagged tokens 1..2 as B-PER, I-PER.
span_tokens = [1, 2]
start = enc["offset_mapping"][span_tokens[0]][0]
end = enc["offset_mapping"][span_tokens[-1]][1]
print(text[start:end]) # 'Ada Lovelace'
# Special tokens like [CLS]/[SEP] carry (0, 0) -- mask before mapping.
for tok_id, (s, e) in zip(enc["input_ids"], enc["offset_mapping"]):
if (s, e) == (0, 0):
continue # skip specials
Real-world examples
Real products, models, and research that use this idea.
1HuggingFace's token-classification and QA pipelines pass return_offsets_mapping=True under the hood so the Rust-backed fast tokenizer can map predicted token spans back to displayed text.
2spaCy's transformer pipelines align wordpiece tokens to spaCy tokens using offset alignment, the same character-span bridging offset_mapping provides.
3Document-AI extraction stacks that ground GPT-5.5 or Claude Opus 4.7 answers back onto a source PDF rely on token to character offsets to draw the highlight box.
4Annotation tools like Prodigy use fast-tokenizer offsets to render model-predicted entity spans onto the raw text a labeler sees.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYour QA model chunks a long document with a sliding window. How do you map an answer span back to the full document?
Sign in or go Pro to see the approach for this follow-up.
QHow do offsets behave when normalization lowercases or strips accents before tokenization?
Sign in or go Pro to see the approach for this follow-up.
QWhen would you still reach for a slow tokenizer despite losing offset_mapping?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Trying to rebuild character spans by string-matching detokenized pieces, which silently breaks on the WordPiece ## prefix and any multi-byte Unicode character.