BERT prepends a [CLS] token to every input, what is its actual job in the model?

MCQ·Easy·4.0 · 0·~1 min·Asked atAlibaba Stability Ai·Relevant atAi4bharat Cerebras Deepseek Microsoft

Attempt it

**The answer is B.** BERT (Bidirectional Encoder Representations from Transformers, 2018) prepends a special `[CLS]` (classification) token to every input. Its job is to act as a **pooled representation slot**. BERT's self-attention is **bidirectional**: every token attends to every other token at every layer. The `[CLS]` token sits at position zero, but because of bidirectionality it can see every real token in the input. Through pretraining (the Next Sentence Prediction task plus the gradient flow from Masked Language Modeling), the model learns to use the `[CLS]` position to **aggregate sequence-level information** into one vector. When you fine-tune BERT for a downstream task like sentiment classification, you do NOT use the full sequence of hidden states. You add a small classification head (a linear layer plus softmax) on top of the **`[CLS]` token's final-layer hidden state**. One vector in, one prediction out. Why the other options are wrong: - **A**: attention masks mark padding, not `[CLS]`. `[SEP]` marks segment boundaries between two sentences. - **C**: BERT uses **learned absolute positional embeddings**, not rotary. RoPE was introduced years later in RoFormer. - **D**: masked tokens in MLM are different tokens marked with `[MASK]`, NOT `[CLS]`. `[CLS]` is never masked.

**B is correct.** `[CLS]` is BERT's pooled representation slot. Because attention is bidirectional in BERT, the `[CLS]` position can attend over every real token in the sequence at every layer. Through pretraining (Next Sentence Prediction directly trains on `[CLS]`, and MLM gradients flow through it indirectly), its final-layer hidden state learns to aggregate sequence-level information. Downstream classification heads take that single `[CLS]` hidden state as input rather than averaging or concatenating all token states: $$\text{logits} = W_\text{cls} \cdot h_\text{[CLS]}^{(L)} + b$$ where `h_[CLS]^(L)` is the final-layer hidden state at position zero. One vector in, one prediction out. **Why the other options are wrong:** - A confuses `[CLS]` with `[SEP]` (segment boundary) or with BOS (autoregressive generation start). Padding is handled by attention masks, not by `[CLS]`. - C invents rotary phase semantics; BERT predates RoPE and uses learned absolute positional embeddings. - D conflates `[CLS]` with `[MASK]`. `[MASK]` is the placeholder used during MLM training to hide tokens for prediction; `[CLS]` is never masked. The `[CLS]` design choice is a standout feature of encoder-only models. Decoder-only models used for embeddings or classification typically use last-token pooling or mean pooling instead. Causal masking prevents a first-position token from seeing later tokens.

TL;DR

The [CLS] token's final-layer hidden state serves as BERT's pooled sequence representation, the single vector fed to downstream classification heads.

Memory aid

Easy to grasp

Imagine a classroom where every student writes a short note about a story they read together. One designated 'class summary' student listens to all the other notes and writes one overall summary of the whole story. When the teacher needs to grade the class on understanding, she just reads the summary student's note instead of all the individual ones. [CLS] is BERT's summary student: it sits at position zero and listens to (attends to) every other token in the sentence. At the end, downstream tasks like sentiment classification or topic detection use its single vector to make a decision.

Key concepts

[CLS] token Fine-tuning BERT Sequence pooling

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The [CLS] token looks superficially like a sentence-boundary marker or like the BOS token from autoregressive LMs. But its actual job is different. It is a learned aggregation slot whose final-layer hidden state serves as the pooled sequence representation for downstream classification.

What [CLS] is and where it sits

BERT prepends the special [CLS] token to every input sequence. The full input format for a single sentence is:

code

[CLS] token_1 token_2 ... token_n [SEP]

For sentence pairs (NLI, question-passage):

code

[CLS] sentence_A_tokens [SEP] sentence_B_tokens [SEP]

What `[CLS]` is not

Not a sequence-boundary marker. [SEP] fills that role.
Not a mask placeholder. [MASK] fills that role during MLM pretraining.
Not a generation start token. There is no autoregressive generation in BERT; the BOS analogy is misleading.
Not a padding marker. The attention mask handles padding, not [CLS].

What `[CLS]` is

It is a learned token embedding (the same embedding lookup table that holds all other token embeddings) at vocabulary ID 101 in BERT. Its embedding is initialized randomly during pretraining setup and is trained end to end to produce a useful pooled representation by the final layer.

Why position zero

It is convention. Because BERT's attention is bidirectional, any position can aggregate information from all other positions. Placing [CLS] at position zero is consistent and easy to extract programmatically (hidden_states[:, 0, :]). It is not structurally required.

Decoder-only models cannot use position-zero pooling because causal attention prevents the first position from seeing later tokens. They use last-token pooling instead.

How bidirectional attention enables pooling

What pretraining teaches [CLS]

Modern evolution and where [CLS] survives today

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

BERT (2018) introduced [CLS] and the pretraining setup. It is still the canonical example studied in NLP courses.
DeBERTa-v3 and ModernBERT continue the [CLS] pooling convention for classification fine-tuning. Both shipped in 2024-2025 production embedding stacks.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does [CLS] pool well even when NSP is removed (RoBERTa)?

MLM gradient flow through bidirectional attention is enough to give [CLS] some aggregation structure. Every masked token's prediction depends on the full context, and gradient signal flows backward through [CLS] because every other position attends to [CLS]. RoBERTa's success showed that direct NSP supervision is not required; the indirect signal from MLM is sufficient for classification fine-tuning to work.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing [CLS] with [SEP] or with autoregressive BOS. [CLS] is a learned aggregation slot for classification; [SEP] separates sentence pairs; BOS in GPT-style models marks generation start.

60 second bullets to scan on the way to the call.

What [CLS] stands for and where it sits in the input sequence
Why bidirectional attention makes position zero a valid pooling slot

Primary sources. Browse if you want the original framing.

What [CLS] is and where it sits

What [CLS] is not

What [CLS] is

Why position zero

What `[CLS]` is not

What `[CLS]` is