Complete the statement about where RLVR gets its reward signal

**RLVR** stands for **Reinforcement Learning with Verifiable Rewards**. The V is the key letter, and the answer to the blank is 'verifiable' (or near-synonyms like 'programmatic', 'automated', 'rule-based'). Ordinary RLHF reward comes from a learned reward model that was trained on human preference pairs. The reward model is itself a neural network, and any neural network has quirks the policy can exploit. Length inflation, hedging, confident phrasing, formatting tricks: these all raise the reward model score without raising true quality. RLVR sidesteps the learned model entirely. A program that knows the correct answer computes the reward: - Math problems: exact match against the gold final answer (or a symbolic equivalence checker). - Code: run the candidate patch against unit tests; reward 1 if all pass, 0 otherwise. - Formal proofs: a proof-checker accepts or rejects. - Structured outputs: a JSON-schema validator accepts or rejects. The reward is binary, deterministic, and unhackable by style. The policy cannot win by padding or bullet-pointing. Only the answer matters. The catch: RLVR only works on tasks where correctness can be checked mechanically. Open-ended writing, empathy, taste, and most conversational tasks have no programmatic checker, so they still need RLHF or RLAIF.

The missing word is **verifiable**. RLVR (Reinforcement Learning with Verifiable Rewards) replaces the learned reward model of RLHF with a programmatic checker that decides whether a rollout is correct. For a math problem: extract the final answer from the rollout, compare it to the gold answer by string or symbolic equivalence. Reward is 1 on match, 0 otherwise. For a coding task: apply the patch, run the unit tests, reward 1 if all pass, 0 otherwise. For formal reasoning: hand the candidate proof to a proof-checker. **The structural advantage** is that the reward is unhackable by style. A learned RM can be flattered by length, format, or hedging because those features correlate with quality on the labelled set. A unit-test runner cannot. The policy's only way to raise reward is to produce a correct answer. **The structural cost** is task scope. RLVR needs a programmatic checker, which only exists for a narrow band of tasks: math with definite answers, code with tests, formal logic, structured outputs. Open-ended writing, conversation, judgement calls, and any tacit-knowledge target have no checker. For those tasks RLHF or RLAIF is still needed. **In practice,** verifiable rewards are a main signal in published reasoning recipes. DeepSeek-R1's reasoning gains came from GRPO over rule-based rewards on math and code. Tulu 3, which named RLVR, trained its RLVR stage with PPO after SFT and DPO. Both combine RLVR with other training for everything else.