Precision mismatch between rollout generation and log-prob/reward scoring injects numerical noise into PPO ratios, which can destabilize policy updates.
Think of grading an exam with two calculators: one rounds aggressively and one keeps full decimals. If practice tests are solved with one calculator but final scores use the other, borderline answers flip unpredictably. PPO has a similar issue when rollouts, log-probs, and rewards are computed under mismatched precision. Small numeric shifts can change update size, making training jittery. Over a long answer, the tiny differences add up, the way a small rounding error on every line of a long bill adds up to a real amount. So teams align precision settings across generation and scoring paths to keep optimization signals consistent.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Modern RL for language models uses two programs that hold the same weights. A fast inference engine (vLLM or SGLang) generates the rollouts, and a training engine (FSDP or Megatron) computes log-probs and gradients. PPO assumes both describe one policy. In floating point they do not: different kernels, batch shapes and precision choices give slightly different probabilities for the same token.
This training-inference mismatch turns on-policy RL into slightly off-policy RL. The gap per token is small. But PPO works through probability ratios and a narrow clip band, and long reasoning responses add up thousands of small gaps. Left alone, it can cause sudden reward collapse that looks like a hyperparameter problem.
Two engines, one policy on paper
A PPO step for a language model has two phases. In the rollout phase, the inference engine samples responses and usually returns the log-probability of each sampled token. In the training phase, the training engine runs the same tokens through the model again to get log-probs it can differentiate, then forms the PPO ratio and the loss.
On paper both phases evaluate the same weights, so they should give the same numbers. In practice they rarely do. The inference engine uses fused attention and sampling kernels, paged KV caches, large and changing batch sizes, and often tensor parallelism. The training engine uses different kernels, a different sharding layout and different reduction orders. Floating-point addition is not associative, so summing the same values in a different order gives a different rounding.
Batch shape matters too. A Thinking Machines study showed that inference kernels are usually deterministic for a fixed batch, but not batch-invariant. A request's numbers change with how many other requests share the batch, and that depends on server load. So even two calls to the same inference engine can disagree, and the trainer almost always disagrees with the sampler.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- A 2025 study of verl-style RL found that a vLLM rollout engine and an FSDP training engine can give noticeably different token probabilities with the same weights. It proposed truncated importance sampling, which verl adopted.
- MiniMax-M1 found a train and inference probability mismatch during RL and raised the LM output head to FP32, lifting the correlation from about 0.9x to 0.99x.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the trainer recomputes the old log-probs itself, the PPO ratio starts at exactly 1. Why is training still affected by the sampler's numerics?
The tokens were drawn from the sampler's distribution, not the trainer's. The update is then off-policy, and the bias needs an importance weight from trainer to sampler.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Many teams debug PPO instability as a hyperparameter issue when the hidden root cause is mismatched numeric precision across rollout and scoring paths.
60 second bullets to scan on the way to the call.
Where the sampler and trainer log-probs come from
Why identical dtypes still give different numbers
Primary sources. Browse if you want the original framing.
- Schulman et al., Proximal Policy Optimization Algorithms
- Yao et al., Your Efficient RL Framework Secretly Brings You Off-Policy RL Training
- Qi et al., Defeating the Training-Inference Mismatch via FP16
- MiniMax, MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
- He, Defeating Nondeterminism in LLM Inference
Same topic, related formats. Practice these next.