When would you need the offset_mapping feature that HuggingFace fast tokenizers provide?
Describe what offset_mapping returns from a HuggingFace fast tokenizer. Give two concrete use cases where offset_mapping is required, and explain what problem arises if you try to implement the same functionality without it using a slow tokenizer.
offset_mapping returns per-token (start, end) character spans, letting you map model predictions back onto the original text for NER, extractive QA, and highlighting.
Imagine highlighting a sentence in a book with sticky tabs, where each tab covers a few letters. Later someone tells you 'the answer is on tab 3 through tab 5', but you need to point at the actual words on the page. You need a little map that says 'tab 3 starts at letter 12 and ends at letter 17'. That map is offset_mapping. The fast tokenizer hands it to you for free; without it you would have to count letters by hand and you would miscount whenever spaces, accents, or emoji get involved.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every transformer NLP system has a seam between two coordinate systems. The model lives in token space: it tags token 3, or predicts that the answer runs from token 18 to token 24. The user lives in character space: they see a string and expect a highlight drawn on the actual words. Something has to translate between the two, and that something is the character offset of each token.
HuggingFace fast tokenizers expose this translation as offset_mapping. It looks like a small convenience, but it is the difference between a span-extraction feature that works on every language and one that silently corrupts output the moment a user types an accented character. This question is really probing whether you understand that token strings are lossy and that recovering positions by hand is a trap.
What the mapping actually contains
When you call a fast tokenizer with return_offsets_mapping=True, each token gets a (start, end) pair of character indices into the input string. Token i corresponds to text[start:end]. For the input Hello world, byte-level BPE produces tokens roughly like Hello and a leading-space world, with offsets 0 to 5 and 5 to 11. The leading space belongs to the second token, and its offset captures that.
The pairs are half-open intervals, the same convention as Python slicing, so concatenating the per-token slices reproduces the original text exactly. That property is what makes the mapping trustworthy: it is computed by the tokenizer as it segments, not reverse-engineered afterward.
Special tokens are the one wrinkle. [CLS], [SEP], and padding carry a zero to zero offset because they correspond to no source characters. Any code that maps predictions back must skip those, or it will think a real prediction landed at position zero.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-cased") # fast by default
text = "Ada Lovelace wrote the first algorithm."
enc = tok(text, return_offsets_mapping=True)
# Suppose the NER model tagged tokens 1..2 as B-PER, I-PER.
span_tokens = [1, 2]
start = enc["offset_mapping"][span_tokens[0]][0]
end = enc["offset_mapping"][span_tokens[-1]][1]
print(text[start:end]) # 'Ada Lovelace'
# Special tokens like [CLS]/[SEP] carry (0, 0) -- mask before mapping.
for tok_id, (s, e) in zip(enc["input_ids"], enc["offset_mapping"]):
if (s, e) == (0, 0):
continue # skip specialsReal products, models, and research that use this idea.
- HuggingFace's token-classification and QA pipelines pass return_offsets_mapping=True under the hood so the Rust-backed fast tokenizer can map predicted token spans back to displayed text.
- spaCy's transformer pipelines align wordpiece tokens to spaCy tokens using offset alignment, the same character-span bridging offset_mapping provides.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour QA model chunks a long document with a sliding window. How do you map an answer span back to the full document?
Use return_overflowing_tokens plus overflow_to_sample_mapping; offsets are per chunk, so add the chunk's start position in the original text.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Trying to rebuild character spans by string-matching detokenized pieces, which silently breaks on the WordPiece ## prefix and any multi-byte Unicode character.
60 second bullets to scan on the way to the call.
What offset_mapping returns per token
Two tasks that need token to character mapping
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.