What is a verifier in RL?

In reinforcement learning, an agent tries a problem and the environment scores the attempt. When that score comes from an automatic check of correctness, the setup is called reinforcement learning with verifiable rewards, or RLVR, a term introduced by Lambert et al. (2024). The verifier is the check itself. Our RLVR guide covers the training method. This page covers the check.

What a verifier checks

A verifier compares what the agent produced with something known to be right. Common forms are

• tests that run against the agent's code change,
• an exact answer checked against source records, such as a figure in a filing or a balance in a ledger,
• the final state of an app or a file system, checked against the state the problem expects,
• checks a reviewer would make, written as code where tests alone miss the point.

In our coding environments, tests are the starting point, and we add the checks a reviewer would make where a problem needs more, written with engineers who know that kind of codebase (/domains/coding). In finance, many questions have a right answer in the source records, and code checks the answer against those records (/domains/knowledge-work). In our own post-training runs, each attempt gets a sandbox with a grader for its problem, and the reward is 1 when the grader passes (post-training stack).

Verifier, grader or model judge

A verifier gives a yes or no, or a number, from rules. A grader is the broader part of an environment that scores an attempt, and it can combine a verifier with a rubric. A model judge, or LLM as a judge, reads the result and scores it, which suits work with no single right answer. Practitioners write the grading criteria for that kind of work, such as a credit memo (/domains/knowledge-work).

Model judges bring their own errors. Our paper Contrastive Decoding for LLM Judges (Findings of ACL 2026) shows that judge scores are highly sensitive to the score range they are given, and reduces that bias for up to an 11.7% relative improvement in correlation with human judgments.

How verifiers fail

A verifier rewards whatever passes it, and an agent in training will find what passes. The gap between what the check covers and what success requires is the verification gap (RLVR guide). Through that gap comes reward hacking. In a coding run, a model that deletes a failing test can earn the same reward as one that fixes the bug (/domains/alignment).

Our TRACE benchmark (ICML 2026) catalogs 54 kinds of reward exploit in code environments across 517 trajectories. The best model tested caught 63% of the hacks when it could compare runs side by side, and 45% when it judged one run alone. Models struggled more with hacks that depend on meaning than with ones visible in the syntax.

How to check a verifier

1. Run it on known-good solutions. We test each grader against worked solutions, so a low score means the model was wrong (/domains/knowledge-work).
2. Have experts review the checks. Practitioners who have done the job write each finance problem and its checks, and separate experts review them before delivery (same source).
3. Read passing runs, not only failing ones. In our post-training experiments we confirmed the model was not reward hacking and that passing runs came from model gains (post-training stack).
4. Compare runs when hunting hacks. TRACE found detection is easier with runs side by side.
5. Look at the process as well as the result. Rewards that check only the outcome can teach the wrong lesson (/domains/alignment).
6. Watch for steps where every attempt fails. When the environment service has an outage, every attempt can fail, rewards come back zero, and the step trains nothing. One of our runs lost 8 of its 50 steps this way (post-training stack).

Beyond pass or fail

A verifier that returns only 1 or 0 limits what a model can learn. Our post-training team is exploring partial credit and rewarding individual steps, since one reward at the end of tens of turns is applied to every turn equally (post-training stack).