When would you need the offset_mapping feature that HuggingFace fast tokenizers provide?

A tokenizer chops your text into tokens and gives back integer IDs, but it throws away one useful thing along the way: where each token sat in the original string. The `offset_mapping` feature puts that information back. When you call a HuggingFace fast tokenizer with `return_offsets_mapping=True`, you get one `(start, end)` pair per token. These are character positions into the input string. Tokenizing `Hello world` gives tokens like `Hello` and ` world` with offsets `(0, 5)` and `(5, 11)`. So token 1 covers characters 0 through 5 of the original text. Why do you care? Because models predict things at the token level, but humans read at the character level. A named-entity model says token 3 is a person name. To draw a highlight in the UI, you need to know which characters token 3 covers. That is exactly what `offset_mapping` tells you. The word fast matters here. Fast tokenizers are written in Rust and track these offsets as they tokenize. The older slow tokenizers do not, so you would have to reconstruct positions yourself, which is fiddly and easy to get wrong.

**offset_mapping** is a per-token list of `(start, end)` character indices into the original input string, returned by HuggingFace fast tokenizers when you pass `return_offsets_mapping=True`. Token i covers `text[start:end]`, so it bridges the gap between what the model predicts (token positions) and what the user sees (characters). Two use cases make it essential. In token classification like NER, the model tags each token, and you map the tagged indices back to character spans to highlight or extract the entity. In extractive QA, the model predicts a start token and an end token, and you recover the answer as `text[offsets[start][0]:offsets[end][1]]`. Without it, on a slow tokenizer, you reconstruct positions by hand. That breaks on WordPiece `##` continuation pieces. It breaks on byte-level BPE where a token is a fragment of a multi-byte character. And it breaks on any Unicode where character index does not equal byte index. Fast tokenizers compute the mapping in their Rust core, so it is exact and free. The slow path is the kind of code that passes your English unit tests and then mangles spans on accented or CJK input in production.