BERT prepends a [CLS] token to every input, what is its actual job in the model?

**The answer is B.** BERT (Bidirectional Encoder Representations from Transformers, 2018) prepends a special `[CLS]` (classification) token to every input sequence. Its job is to act as a **pooled representation slot**. Here is the mechanic. BERT's self-attention is **bidirectional**: every token attends to every other token at every layer. The `[CLS]` token sits at position zero, but because of bidirectionality it can see every real token in the input. Through pretraining (specifically the Next Sentence Prediction task plus the gradient flow from Masked Language Modeling), the model learns to use the `[CLS]` position to **aggregate sequence-level information** into one vector. When you fine-tune BERT for a downstream task like sentiment classification, NLI, or topic classification, you do NOT use the full sequence of hidden states. You add a small classification head (a linear layer plus softmax) on top of the **`[CLS]` token's final-layer hidden state**. One vector in, one prediction out. Why the other options are wrong: - **A**: padding is marked by attention masks (`attention_mask = 0` for padding positions), not by `[CLS]`. The `[SEP]` token marks segment boundaries between two sentences. - **C**: BERT uses **learned absolute positional embeddings**, not rotary. RoPE was introduced years later in RoFormer and adopted by GPT-NeoX, Llama, etc. - **D**: masked tokens in MLM are different tokens marked with `[MASK]`, NOT `[CLS]`. `[CLS]` is never masked.

**B is correct.** `[CLS]` is BERT's pooled-representation slot. Because attention is bidirectional in BERT, the `[CLS]` position can attend over every real token in the sequence at every layer. Through pretraining (Next Sentence Prediction directly trains on `[CLS]`, and MLM gradients flow through it indirectly), its final-layer hidden state learns to aggregate sequence-level information. Downstream classification heads take that single `[CLS]` hidden state as input rather than averaging or concatenating all token states: $$\text{logits} = W_\text{cls} \cdot h_\text{[CLS]}^{(L)} + b$$ where `h_[CLS]^(L)` is the final-layer hidden state at position zero. One vector in, one prediction out. **Why the distractors fail:** - A confuses `[CLS]` with `[SEP]` (segment boundary) or with BOS (autoregressive generation start). Padding is handled by attention masks, not by `[CLS]`. - C invents rotary phase semantics; BERT predates RoPE and uses learned absolute positional embeddings. - D conflates `[CLS]` with `[MASK]`. `[MASK]` is the placeholder used during MLM training to hide tokens for prediction; `[CLS]` is never masked. The `[CLS]` design choice is a standout feature of encoder-only models. Decoder-only LLMs (GPT-5.5, Claude Opus 4.7) typically use last-token pooling or mean pooling instead, since causal masking prevents a first-position token from seeing later tokens.