BERT prepends a [CLS] token to every input, what is its actual job in the model?

MCQ·Easy·4.0 · 0·~1 min

Attempt it

**The answer is B.** BERT (Bidirectional Encoder Representations from Transformers, 2018) prepends a special `[CLS]` (classification) token to every input sequence. Its job is to act as a **pooled representation slot**. Here is the mechanic. BERT's self-attention is **bidirectional**: every token attends to every other token at every layer. The `[CLS]` token sits at position zero, but because of bidirectionality it can see every real token in the input. Through pretraining (specifically the Next Sentence Prediction task plus the gradient flow from Masked Language Modeling), the model learns to use the `[CLS]` position to **aggregate sequence-level information** into one vector. When you fine-tune BERT for a downstream task like sentiment classification, NLI, or topic classification, you do NOT use the full sequence of hidden states. You add a small classification head (a linear layer plus softmax) on top of the **`[CLS]` token's final-layer hidden state**. One vector in, one prediction out. Why the other options are wrong: - **A**: padding is marked by attention masks (`attention_mask = 0` for padding positions), not by `[CLS]`. The `[SEP]` token marks segment boundaries between two sentences. - **C**: BERT uses **learned absolute positional embeddings**, not rotary. RoPE was introduced years later in RoFormer and adopted by GPT-NeoX, Llama, etc. - **D**: masked tokens in MLM are different tokens marked with `[MASK]`, NOT `[CLS]`. `[CLS]` is never masked.

**B is correct.** `[CLS]` is BERT's pooled-representation slot. Because attention is bidirectional in BERT, the `[CLS]` position can attend over every real token in the sequence at every layer. Through pretraining (Next Sentence Prediction directly trains on `[CLS]`, and MLM gradients flow through it indirectly), its final-layer hidden state learns to aggregate sequence-level information. Downstream classification heads take that single `[CLS]` hidden state as input rather than averaging or concatenating all token states: $$\text{logits} = W_\text{cls} \cdot h_\text{[CLS]}^{(L)} + b$$ where `h_[CLS]^(L)` is the final-layer hidden state at position zero. One vector in, one prediction out. **Why the distractors fail:** - A confuses `[CLS]` with `[SEP]` (segment boundary) or with BOS (autoregressive generation start). Padding is handled by attention masks, not by `[CLS]`. - C invents rotary phase semantics; BERT predates RoPE and uses learned absolute positional embeddings. - D conflates `[CLS]` with `[MASK]`. `[MASK]` is the placeholder used during MLM training to hide tokens for prediction; `[CLS]` is never masked. The `[CLS]` design choice is a standout feature of encoder-only models. Decoder-only LLMs (GPT-5.5, Claude Opus 4.7) typically use last-token pooling or mean pooling instead, since causal masking prevents a first-position token from seeing later tokens.

Quick reference

TL;DR

The [CLS] token's final-layer hidden state serves as BERT's pooled sequence representation, the single vector fed to downstream classification heads.

Memory aid

Easy to grasp

Imagine a classroom where every student writes a short note about a story they read together, and one designated 'class summary' student listens to all the other notes and writes one overall summary of the whole story. When the teacher needs to grade the class on understanding, she just reads the summary student's note instead of all the individual ones. [CLS] is BERT's summary student: it sits at position zero, listens to (attends to) every other token in the sentence, and at the end its single vector is what downstream tasks like sentiment classification or topic detection use to make a decision.

Key concepts

[CLS] token Fine-tuning BERT Sequence pooling

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: [CLS] as a learned pooled-representation slot, how bidirectional attention enables position-zero pooling, role of NSP and MLM training, why classification heads consume only [CLS] hidden state, distinction from [SEP] and [MASK], contrast with decoder-only last-token pooling.

Real-world examples

Real products, models, and research that use this idea.

BERT (2018) introduced [CLS] and the pretraining recipe; still the canonical example studied in 2026 NLP courses.
DeBERTa-v3 and ModernBERT continue the [CLS] pooling convention for classification fine-tuning, both shipped in 2024-2025 production embedding stacks.
Cross-encoder rerankers like the bge-reranker family take the [CLS] hidden state as their relevance score head, used widely in 2026 RAG stacks alongside Llama 4 Maverick and Claude Opus 4.7.
Sentence-BERT replaced [CLS] pooling with mean pooling for sentence embedding tasks; the trade-off depends on whether you need classification or retrieval geometry.
Decoder-only LLMs (GPT-5.5, Llama 4) skip [CLS] entirely because causal attention prevents position-zero from seeing later tokens; they use last-token pooling instead.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does [CLS] pool well even when NSP is removed (RoBERTa)?

QHow does Sentence-BERT improve sentence embedding quality over vanilla BERT's [CLS]?

QCould a decoder-only LLM be retrofitted with a [CLS] token for classification?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing [CLS] with [SEP] or with autoregressive BOS. [CLS] is a learned aggregation slot for classification; [SEP] separates sentence pairs; BOS in GPT-style models marks generation start.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.