Fill the two blanks in the standard reward-model architecture statement

Most reward models reuse the same transformer backbone as the policy family. The only architectural addition is a small output head that maps the final representation to a single scalar. That scalar represents estimated preference quality for the whole completion. This is different from the policy model, which predicts next-token distributions. Reward models are sequence scorers, not text generators. The scalar head is enough because training labels are usually pairwise choices, not token-level supervision. So in the fill-blank statement, the first blank points to a scalar or reward head, and the second blank points to a single reward-like score.

The canonical architecture is: reuse pretrained transformer layers, add a scalar-valued reward head, and predict one quality score per completion. That score is used for ranking chosen versus rejected outputs. In equation form, the model outputs `r(x, y)` for prompt `x` and completion `y`, then pairwise training uses score differences rather than token-level likelihood. The practical reason this architecture is common is implementation simplicity: you preserve language understanding capacity and only learn a compact scoring surface for preference optimization.