Special Tokens

Defend pinning the tokenizer revision alongside the model weights when shipping a fine-tune

Qwen 3.5 SFT data: fill the special tokens that wrap an assistant turn.

Define the role of a chat template when fine-tuning on (system, user, assistant) data

A chat template wraps every turn with BOS and EOS tokens. Why does the model need them?

After editing the Jinja chat template, your fine-tuned Llama 3 prints raw special token text. What broke?

$50K compute budget, 50B tokens of US legal text, 1.5B parameter model. Defend your tokenizer design.

Spot the bug: a Llama-3 fine-tune that skips the end of turn token

Why is chat template integrity the most common 'silent quality killer' in FT?

PremiumDescribe how a Special…

Which tokenizer parameter prevents user text from being interpreted as special tokens?

What are BOS, EOS, PAD, and role marker special tokens, and what happens when each is missing or duplicated?

Fill in the blanks: the special tokens that control sequence boundaries and conversation structure in modern LLMs.

What failure mode occurs when a developer duplicates the BOS token by prepending it manually while also using apply_chat_template?

PremiumWhy must developers use…