How to Evaluate RL Environments Before You Buy

01 Task realism

Tasks come from the work your agent will do, in the software it will use.

Widely used agent benchmarks build their tasks from real settings. SWE-bench draws 2,294 problems from real GitHub issues and pull requests across 12 Python repositories. WebArena built fully functional websites in four domains because agents tested in simplified synthetic environments showed “a disconnect with real-world scenarios”. OSWorld derives each of its 369 tasks from real computer use cases in real web and desktop apps.

What to ask

• Where do the tasks come from, and who wrote them?
• Does the agent work in the real software, or a replica? If a replica, which behaviors does it reproduce and which does it leave out?
• Do the tasks span several steps and apps, as the work does?

Sources [1] [2] [3]

02 Verifiers that measure success

A passing score means the task was done, not that a weak check was satisfied.

A verifier decides whether the agent succeeded, and it can be wrong. The authors of the Agentic Benchmark Checklist call this outcome validity, “the evaluation result (e.g., tests or checks) truly indicates task success”. They found flaws in outcome validity in seven of the ten agentic benchmarks they assessed, and report that such issues can under- or overestimate agent performance by up to 100% in relative terms.

Weak tests are one cause. In a manual review of SWE-bench patches that SWE-Agent with GPT-4 got to pass, SWE-Bench+ found 31.08% were suspicious because the tests were not adequate to verify the patch.

What to ask

• How is each verifier tested before delivery? Ask for its results on known wrong solutions.
• Which tasks use a model as the grader, and how is that grader checked?
• Can you read the verifier code or rubric for a sample of tasks?

Sources [4] [5]

03 Resistance to reward hacking

The environment has been attacked by its builder, and the exploits found were closed.

In RL the model is trained to raise the score, so any gap between the score and the goal becomes something it learns. Concrete Problems in AI Safety lists avoiding reward hacking among five practical research problems. Defining and Characterizing Reward Hacking defines it as optimizing an imperfect proxy reward that leads to poor performance on the true reward, and shows that a proxy that can never be hacked is a very strong condition.

It also shapes training choices. The DeepSeek-R1 paper says its team did not use neural reward models for DeepSeek-R1-Zero “because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process”, and used rule-based rewards instead (section 2.2.2). Our own paper Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis studies how to detect reward hacks in code environments.

What to ask

• What exploits has the builder tried against its own graders, and what did it fix?
• Can the agent read or change the tests, the grader or the expected answers from inside the environment?
• Are training runs audited for runs that score well without doing the task?

Sources [6] [7] [8]

04 No shortcuts to a passing score

A task is solvable only by an agent that has the skill, and an empty answer never passes.

The Agentic Benchmark Checklist calls this task validity, “a task should be solvable if and only if the agent possesses the target capability”. Its authors found that in τ-bench a trivial agent that returns empty responses is counted as successful on intentionally impossible tasks, reaching a 38% success rate.

Answers can also leak into the task. SWE-Bench+ found that 32.67% of successful SWE-bench patches had the solution provided in the issue report or its comments, which its authors call solution leakage.

What to ask

• Has the builder run a do-nothing agent and an empty answer against every task?
• Is the answer, or a strong hint, anywhere in the task’s inputs, files or history?
• For tasks meant to be refused or impossible, what exactly earns the reward?

Sources [4] [5]

05 Contamination

The tasks and answers are not already in the model’s training data.

Tasks a model has seen in pretraining measure memory, not skill. GSM1k, a fresh set built to mirror GSM8k, found accuracy drops of up to 8% for leading models, with evidence that some had partly memorized GSM8k. SWE-Bench+ reports that over 94% of SWE-bench issues were created before the models’ knowledge cutoff dates.

Builders guard against it in two ways. LiveCodeBench keeps collecting new problems from coding contests over time to keep its evaluation free of contamination. Tulu 3 reports substantial decontamination of existing open datasets against the benchmarks it evaluates on.

What to ask

• Are tasks written new, or adapted from public material? If public, from when?
• Has the set been checked for overlap with the benchmarks you report on?
• Will the tasks stay private, or are they published?

Sources [9] [5] [10] [11]

06 Diversity and held-out tasks

There are enough distinct tasks to learn a skill, and a split the model never trains on.

Procgen Benchmark found that diverse environment distributions are essential to train and evaluate RL agents adequately. Quantifying Generalization in Reinforcement Learning notes that using the same environments for training and testing offers little insight into generalization, and found that agents overfit to surprisingly large training sets.

What to ask

• How many distinct tasks are there, and how many templates or seeds are they built from?
• Is there a held-out test split, and how is it kept apart from training?
• How are near-duplicate tasks found and removed?

Sources [12] [13]

07 Difficulty calibration

Tasks sit where the model you are training sometimes succeeds and sometimes fails.

A task every rollout solves, or none does, teaches little in group-based RL. The DAPO authors over-sample and filter out prompts with accuracy equal to 1 and 0, so that every prompt in a batch has an effective gradient (section 3.2). That makes per-task pass rates on the model in training a basic thing to ask for, often reported as pass@k.

Difficulty can also be measured in human terms. Measuring AI Ability to Complete Long Software Tasks times people with domain expertise on each task and reports the length of task a model completes with 50% success. In the first version of the WebArena paper (Jul 25, 2023), humans succeeded on 78.24% of tasks against 10.59% for the best GPT-4-based agent, showing how much room a task set can leave.

What to ask

• What share of tasks does the model you are training solve today, task by task?
• How long does each task take a skilled person?
• Can difficulty be adjusted as the model improves?

Sources [14] [15] [2]

08 Reliability across repeated runs

Scores are measured over several runs, and the environment behaves the same each time.

One success can hide an agent that fails most of the time. τ-bench introduced pass^k, the chance that an agent succeeds on every one of k tries, and found agents “quite inconsistent”, with pass^8 below 25% in its retail domain.

What to ask

• Do results come from one run per task, or several?
• Does the environment give the same result for the same actions, apart from intended randomness?
• Does the grader give the same verdict when run twice?

Sources [16]

09 Reproducible state

Every run starts from the same known state and can be rerun anywhere.

OSWorld gives each task a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. WebArena set out to build an environment that is both highly realistic and reproducible.

What to ask

• Can the environment be reset to an exact starting state for each run?
• Does it depend on live websites or outside services that can change?
• How is it packaged and run, and on what infrastructure?

Sources [3] [2]

10 Licensing and provenance

You know where every task came from and what you may do with it.

The Data Provenance Initiative audited more than 1,800 text datasets and found frequent miscategorization of licenses on widely used dataset hosting sites, with license omission above 70% and error rates above 50%.

What to ask

• Who owns the tasks, the graders and the runs your agents produce in them?
• What third-party content is inside, such as code, documents or websites, and under what licenses?
• Is there a written record of where each task came from and who made it?

Sources [17]

11 Documentation of limits

Known issues are written down, so you can account for them.

Some evaluation issues cannot be fully avoided, especially with model graders or test-based checks. The Agentic Benchmark Checklist asks builders in that case to be transparent and clearly communicate the impact of those limits, and found limitations in result reporting in all ten benchmarks it assessed.

What to ask

• Is there a list of known issues per task set, and how often is it updated?
• How are bad tasks reported, fixed and replaced after delivery?

Sources [4]