Problem process reward models solve that outcome only rewards miss.

In reasoning RL, an **outcome reward** checks only the final answer — correct extraction, passing unit tests, or a proof checker at end-of-sequence. On a long chain-of-thought with dozens of steps, that is one sparse signal for the entire trajectory. Two failure modes follow. First, a model may take an invalid intermediate step, wander, then self-correct and still get full reward — the bad step was never penalized. Second, a mostly correct derivation with a small final slip gets the same punishment as nonsense, giving no gradient on the good early reasoning. **Process reward models (PRMs)** score intermediate steps — each line, equation, or labeled reasoning segment gets a correctness label. RL can assign credit along the chain: reinforce sound steps, discourage flawed logic even when the final answer is accidentally right. PRMs address the **credit assignment problem** on long CoT that outcome-only verifiers cannot see.

Outcome-only rewards apply a single check at the end of a long reasoning trace. That sparsity creates ambiguous credit assignment on chain-of-thought RL. **What outcome rewards miss:** - Wrong intermediate steps that get self-corrected still receive full EOS reward — the policy learns that sloppy exploration can work. - Correct reasoning with a final arithmetic slip is punished uniformly — no partial credit for good process. - Lucky guesses and genuine derivations look identical to the verifier if the final answer matches. **What PRMs add:** step-level labels (often +/− per reasoning segment) so the reward function sees *how* the model got there. Training can prefer trajectories with consistently positive step scores, penalize invalid leaps, and improve sample efficiency on math and logic. PRMs do not replace outcome verifiers — they complement them. Production stacks often combine EOS correctness with process scores, or use implicit process signals in GRPO variants that compare diverse rollouts.