Process reward models: grading the reasoning, not just the answer
OpenAI's process-supervision research shows that rewarding every correct reasoning step, not only the final answer, catches errors outcome supervision misses and produces a more reliable verifier.
AI-assisted / research-based
This field note was drafted with AI assistance and synthesizes publicly available research papers and disclosed industry practice on an emerging AI technique. It is not based on confidential deployment data, is not investment, legal, medical, or security advice, and every primary claim links directly to its source so you can verify it yourself.
A model can reach a correct final answer through flawed reasoning, and it can reach an incorrect answer despite a mostly sound approach. A reward model trained only on final-answer correctness cannot distinguish these cases — and in a multi-step task, that blind spot is exactly where errors compound.
Grading the answer tells you whether the model got lucky. Grading the reasoning tells you whether it can be trusted next time.
Two ways to supervise the same reasoning trace
Let's Verify Step by Step ↗, by Lightman and colleagues at OpenAI, directly compares two reward-model training regimes on the same underlying task. An outcome-supervised reward model is trained only on whether a solution's final answer was correct. A process-supervised reward model is trained on human-labeled correctness judgments for each individual reasoning step in a solution — a much denser, more specific training signal per example.
The reward model sees every step, not just the ending
The paper's central finding is that process supervision significantly outperforms outcome supervision on the challenging MATH dataset: the process-supervised model solves 78% of problems from a representative test subset, and active learning — prioritizing which steps get human labels — further improves the efficacy of process supervision per unit of labeling effort. The team released PRM800K, a dataset of 800,000 step-level human feedback labels, specifically to support further research on this distinction.
Why step-level scoring changes what a reward model can do
A reward model that scores only completed answers can only ever accept or reject a finished solution. A process reward model can score a partial reasoning trace before it finishes — which changes what it is useful for in a production pipeline:
- It can rank multiple in-progress candidate solutions, allowing a search procedure to prune unpromising reasoning paths before they run to completion, rather than generating full solutions and comparing only at the end.
- It can localize an error to the specific step it occurred at, rather than only flagging that the overall solution is wrong — directly useful for debugging a reasoning pipeline or explaining a rejection to a reviewer.
- It can supply denser training signal per example than a single pass/fail label, which the paper's active-learning results suggest makes each unit of human labeling effort go further.
Score every step, not just the ending
The cost this trades against
Step-level human labels are more expensive to produce than a single correctness label per solution — a human reviewer has to read and judge every intermediate step, not just check a final number. This cost is not incidental; it is the central practical trade-off the paper's active-learning strategy is designed to reduce, by prioritizing which steps most need a human label rather than labeling every step of every solution uniformly.
The definition of "a correct step" is also domain-specific. What counts as valid progress in a MATH-style proof does not directly transfer to a legal argument, a clinical differential, or a code review — a process reward model trained on one domain's notion of valid reasoning cannot be assumed to generalize to another's without new domain-specific labels.
What this changes in production
Reasoning-heavy verification pipelines
Math, code generation, and multi-step analytical tasks where an incorrect final answer is expensive can use a process reward model as a pre-delivery check — scoring the reasoning trace, not just pattern-matching the final answer format, before a result reaches a user or a downstream automated system.
Best-of-N candidate selection
Generating several candidate reasoning traces for a hard question and using a process reward model to select the most credible one is a direct, practical use of step-level scoring — and one where the paper's finding that process supervision outperforms outcome supervision for selection tasks specifically applies.
Training signal for reasoning-focused fine-tunes
Where a team is fine-tuning a model specifically for multi-step reasoning tasks, step-level feedback provides a denser signal per training example than a single outcome label, at the cost of more expensive data collection — a trade that active learning can partially offset.
A production checklist
Before relying on a process reward model in production, the team should be able to answer:
- Was the reward model trained on step-level labels specific to this domain, or borrowed from a different reasoning domain where "a correct step" means something different?
- How is a partial-trace score used — to prune search, to gate a final answer, or both — and is that use case actually what the reward model was validated for?
- What is the cost and cadence of collecting new step-level labels as the underlying task or model changes?
- Does the reward model's own judgment get periodically checked against fresh human review, or does it operate unaudited indefinitely?
- Is the reward model being used to select among candidates, to train a policy, or both — and does its reported accuracy hold for the specific use case?
- What happens when the reward model itself is uncertain — does the pipeline have an escalation path, or does it force a selection regardless?
Research referenced
- Lightman et al. (2023), Let's Verify Step by Step ↗.