Resources

An FAQ on RL environments for training AI agents

Patronus AI teamAs of November 202614 min read

Patronus AI publishes this FAQ and builds RL environments. Every claim links its source, read on Nov 18, 2026.

RL environments are the tasks, software and graders that AI labs use to train agents with reinforcement learning. Public reporting from Epoch AI, SemiAnalysis and TechCrunch says labs build them in house, buy them from startups and human data companies, and work with product companies on their software, at prices that run from hundreds of dollars per task to six or seven figures per quarter. Coding leads demand, enterprise workflows are a major growth area, and the people Epoch AI interviewed named resistance to reward hacking as the most important quality criterion.

Snapshot as of November 2026. Last updated

What is an RL environment?

An RL environment is the setting an AI model acts in during reinforcement learning. It gives the model a task, a set of actions it can take and a way to score the result. Jean-Stanislas Denain and Chris Barber wrote Epoch AI’s FAQ on RL environments (Jan 12, 2026) from interviews with 18 people at environment startups, neolabs and frontier labs. It defines the environment as the actions the model can take and the surrounding context that sets their effect, such as environment variables, file systems or the state of a simulated application. Each task is a prompt with an objective and a grader that decides whether, or how far, the objective was met. The authors note that the terms are not fully standardized and that the line between environment and task is fuzzy.

Published benchmarks are built the same way. SWE-bench gives a model 2,294 problems drawn from real GitHub issues and pull requests across 12 Python repositories, and its evaluation applies the model’s patch and runs the unit and system tests tied to each task. OSWorld gives an agent 369 tasks in real web and desktop apps, each with its own starting state and evaluation script. Prime Intellect, which runs an open hub for environments, describes them as defining “the world, rules and feedback loop of state, action and reward” (Aug 27, 2025).

In practice, Epoch AI reports, an environment is often delivered as a Docker container. A computer use environment such as a spreadsheet clone can carry hundreds of tasks, while a coding environment often holds one task per repository state. For the vocabulary, see our guide to RL environments and the glossary entries for verifiers and simulated apps.

Sources [1] [8] [9] [5]

How do AI labs use RL environments?

Labs use them mostly for reinforcement learning. The model attempts a task many times, the grader scores each attempt and the scored attempts update the model’s weights. Epoch AI’s interviewees named two other uses, benchmarking and supervised fine-tuning (SFT) on trajectories that solve the task. One employee of an environment startup told Epoch AI that requests for RL outnumber requests for benchmarking environments by perhaps 10 to 20 times.

Epoch AI dates the current wave to OpenAI’s o1, which was trained on math and coding problems with verifiable answers, and notes that labs have since widened the range of tasks they train on. The Tulu 3 authors (Nov 22, 2024) named the method Reinforcement Learning with Verifiable Rewards (RLVR). The DeepSeek-R1 paper (Jan 22, 2025) reports that DeepSeek-R1-Zero, “a model trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT) as a preliminary step, demonstrates remarkable reasoning capabilities.” Our guide chapter on RLVR covers the method.

Labs also reuse what environments produce. SemiAnalysis (Jan 6, 2026) writes that website replicas bought for agent training are reused for future models, and that trajectories and logs from earlier runs are fed back into stages such as mid-training. Epoch AI adds that SFT suits cases where good trajectories are easy to produce but a reliable grader is hard to build, and that one founder found trailing labs more interested in SFT than in RL.

Sources [1] [6] [7] [2]

Who builds RL environments for AI labs?

Labs build many environments themselves. Epoch AI cites job postings at xAI and Anthropic for in-house environment work and one founder who told the authors there had been “substantially more in-housing”. In September 2025 Jennifer Li of Andreessen Horowitz told TechCrunch that “All the big AI labs are building RL environments in-house”, and that labs also look to outside vendors because the work is complex.

Labs also buy from specialized startups. SemiAnalysis counted “More than 35 companies” built to sell RL environments as of Jan 6, 2026, some cloning websites and others building replicas of tools such as Slack, Salesforce and Gmail. Prime Intellect opened its work to others with the Environments Hub, launched on Aug 27, 2025 as an open, community platform. SemiAnalysis also describes HUD as building tooling that wraps software in containers for use as environments.

Human data companies sell them too. Epoch AI names Mercor, Surge, Handshake and Turing as data providers that now also sell RL environments, and quotes one founder who said their main value is that “they have the guys”, the staff to scale task creation quickly. Epoch AI thinks it likely that product companies already partnering with labs on their own software are often at least collaborating on environment or task creation.

Research labs also build environments and publish the methods behind them. Patronus AI is one of these labs. It builds RL environments for alignment, coding, knowledge work, science and computer use (Domains), and on Jun 25, 2026 it announced a $50M Series B led by Greenfield Partners. Our dated lists cover the field company by company, in the RL environment startups landscape, the RL environment companies roundup, the market map and the downloadable dataset.

Sources [1] [3] [2] [5] [21] [22]

How much do RL environments cost?

Public figures are scarce and come from interviews and press reports, so treat them as ranges with dates. Epoch AI (Jan 12, 2026) reports contracts that are often six to seven figures per quarter, and a neolab researcher who had seen contracts of $300,000 to $500,000. Its interviewees put most tasks at $200 to $2,000 each, with $20,000 for a task rare but possible.

Environment prices vary with fidelity. SemiAnalysis (Jan 6, 2026) reports that website replicas, which it calls “UI gyms”, often cost about $20,000 per website. One of Epoch AI’s interviewees cited figures around $300,000 for high-quality replicas of complex products such as Slack. Two founders told Epoch AI that exclusive deals cost roughly 4 to 5 times as much as non-exclusive ones.

Mechanize (Aug 18, 2025) compared task prices with training compute. It projected $480 of compute per RL task per training run, or $2,400 over five reuses, and argued that cheap tasks waste that compute. On total spending, TechCrunch reported in September 2025, citing The Information, that leaders at Anthropic had discussed spending more than $1 billion on RL environments over the next year. Patronus AI’s own prices are not part of this FAQ.

Sources [1] [2] [4] [3]

What domains do RL environments cover?

Math and coding came first. The Epoch AI authors note that math tasks need only verifiable answers, not a complex environment. One interviewee said math might be shrinking, and coding remains a major source of demand, with coding environments moving from SWE-bench Verified-style tasks toward production setups with GitHub, Linear and a code IDE. SemiAnalysis (Jan 6, 2026) calls coding environments the highest in demand.

In Epoch AI’s reporting, enterprise workflows such as filing an expense report, building a pivot table or updating records in a CRM are a major growth area. These run as tool integrations, browser actions or screenshot-based computer use, often on clones of apps such as Slack or SAP. SemiAnalysis adds environments for chip design, accounting and electronic health records, and reports rising spending on non-professional areas such as photography, music and design. Epoch AI’s interviewees also pointed to longer tasks that span several tabs and browsers.

Patronus AI builds environments for alignment, coding, knowledge work, science and computer use, and each domain has its own page. Its benchmark MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments is one example of a multi-app setting. Its abstract describes “integrating asynchronous events across multiple communication and productivity platforms such as Slack, Linear and Git.” Our roundups list builders by domain for coding, computer use and knowledge work.

Sources [1] [2] [21] [16]

Are RL environments graded by verifiers or LLM judges?

Builders use both. Epoch AI describes the grader as typically automated, “such as a unit test or an LLM judging against a rubric.” A rule-based verifier checks the result with code, for example by running tests or comparing an answer. OpenAI’s grader documentation shows the range, from string check graders that return 0 or 1 to score model graders that return a numeric score from a model prompt.

The DeepSeek-R1 authors used a rule-based reward for DeepSeek-R1-Zero and wrote that they did not use a neural reward model “because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process” (section 2.2.2). Model judges and rubrics cover tasks where many answers pass a test. A researcher quoted by Epoch AI said of coding tasks, “There’s probably many solutions that pass the unit tests. But there’s some nicer solutions that have better trade-offs in terms of your engineering design decisions.”

Either kind can be wrong. The authors of the Agentic Benchmark Checklist found flaws in outcome validity in seven of the ten agentic benchmarks they assessed. Outcome validity asks whether a passing result means the task was done. In a manual review of SWE-bench patches that SWE-Agent with GPT-4 got to pass, SWE-Bench+ found 31.08% suspicious because the tests were not adequate to verify them. Some builders combine both kinds. Patronus AI’s GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations compares each prediction with an execution-derived reference “using a hybrid architecture with both deterministic gates and a semantic judge.”

Sources [1] [10] [7] [11] [12] [17]

What is reward hacking in RL environments?

Reward hacking is when a model raises its score without doing the task. Skalse et al. define it as optimizing an imperfect proxy reward in a way that leads to poor performance on the true reward. Epoch AI’s interviewees named robustness against it as the most important quality criterion. One neolab researcher gave examples of a model “searching up a solution” or “checking out future commits” in a coding repository, and an environment founder said “It takes many many iterations to check against reward hacking.”

One defense is to run a trivial agent against every task. The Agentic Benchmark Checklist found that in τ-bench a trivial agent that returns empty responses is counted as successful on intentionally impossible tasks, reaching a 38% success rate on the airline tasks. Another is a rule-based reward where one is possible, as the DeepSeek-R1 team chose. A third is to screen training runs for exploits with a model as the detector. The TRACE authors write that “As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied.”

Patronus AI studies this third defense in Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis, a main conference paper at ICML 2026. Its abstract proposes “a novel taxonomy of reward exploits spanning across 54 categories” and introduces TRACE, “a synthetically curated and human-verified benchmark containing 517 testing trajectories.” It reports that “models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best detection rate at 63%, up from 45% in isolated settings on TRACE.” Our alignment page covers environments built around this problem.

Sources [13] [1] [11] [7] [15]

How difficult should RL tasks be for training?

Tasks should be hard enough that the model sometimes fails and sometimes succeeds. If every attempt passes or every attempt fails, group-based RL methods get no signal about which approach worked. The DAPO authors filter out prompts with accuracy equal to 1 and 0 so that every prompt in a batch gives an effective gradient. Per-task pass rates on the model being trained, often reported as pass@k, show where each task sits.

Several of the people Epoch AI interviewed wanted a minimum pass rate of around 2 to 3%, or at least one success in 64 or 128 rollouts. One suggested a mix of tasks at 0%, 5% and 30%, since 0% tasks can become learnable after some training, and dropping a task once it reaches around 70%. A founder warned that a task at 0% may be infeasible unless another annotator has solved it blind.

Fixed task sets lose value as the model improves. The abstract of Patronus AI’s position paper Adaptive Environments For LLM Agents Need to Jointly Scale Task, Scaffolding and Reward Design says that continuous training or hill-climbing through repeated evaluation “is bound to saturate” static datasets, hand-written environments and fixed benchmarks, and argues for “jointly scaling all three components”, the tasks, the tools and the reward design.

Sources [14] [1] [19]

What comes with an RL environment when a lab buys one?

Across these sources, a delivered environment has a runnable sandbox, a set of tasks, a starting state for each task, a grader for each task and often reference solutions. OSWorld gives each task “a detailed initial state setup configuration” and a custom execution-based evaluation script. SemiAnalysis reports that domain contractors design tasks, write expected solutions, specify reward signals as rubrics or strict verifiers and in some cases grade model solutions.

Some purchases also include trajectories. Epoch AI defines gold trajectories as human-verified examples of completing a task, used as training data for supervised fine-tuning. For computer use, the Patronus AI Domains FAQ says that expert trajectories can show how a skilled person moves through a task.

Labs can also buy parts on their own. The same FAQ says simulated apps, mock websites and the data behind them can be bought separately, and that Patronus AI environments go through manual reviews by domain experts and automated checks before delivery. Ownership of the data from a custom build is set in each agreement, and how tasks run in a buyer’s own training harness is confirmed when the work is scoped.

Sources [9] [2] [1] [21]

How can a lab tell whether an RL environment is good?

Ask for evidence on a sample. Check that tasks are realistic, that the graders reject known wrong answers, that the environment has been tested for reward hacking, that an empty answer never passes, that tasks are not already in public training data and that difficulty fits the model being trained. Our buyer’s checklist, How to Evaluate RL Environments Before You Buy, turns eleven such checks into questions, each with the research behind it.

The Epoch AI interviewees called scaling task creation without losing quality the main operational bottleneck. One founder said “Finding the experts isn’t that hard, but managing them and doing quality control is hard.” A lab researcher warned about app clones and said “what everyone does is vibe code a buggy website which isn’t useful.” The Agentic Benchmark Checklist also found limits in how results were reported in all ten benchmarks it assessed, so ask for a written list of known issues.

Sources [1] [11]

Can RL environments be generated or simulated automatically?

Some parts can. Coding tasks are already mined and synthesized at scale. SemiAnalysis describes the SWE-rebench pipeline, which filtered 450,000 pull requests down to 21,336 tasks, and SWE-smith, which synthesizes bugs in working code, in part by prompting a model to introduce them. It also reports that DeepSeek used 24,667 coding tasks extracted from GitHub to train V3.2. Mechanize (Aug 18, 2025) frames the choice as quality against quantity, where procedurally generated tasks cost less to make but give weaker reward signals.

A newer approach uses a language model as the environment. The model predicts how the environment’s state changes after each action. The abstract of Patronus AI’s Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL, a NeurIPS 2026 main conference paper, says that world models that simulate environment states “have matched pure rollout performance, making them promising for scaling diversity on-demand.” GlobeBench, also from Patronus AI, is a benchmark of 1,632 tasks across five domains that tests how well models make those predictions.

Patronus AI’s Generative Simulators post (Dec 17, 2025) describes “adaptive environments that jointly co-generate tasks, world dynamics, and reward functions.” Generated environments still need the same checks as hand-built ones, including a grader that holds up against reward hacking.

Sources [2] [4] [18] [17] [20] [1]

About this FAQ

Patronus AI publishes this FAQ. It builds RL environments and is one of the kinds of company it describes, so its research appears as examples, named by each paper’s full title. The format follows Epoch AI’s FAQ on RL environments, which this page cites throughout.

The Sources list gives the full reference for every numbered source. The page is not updated as the field moves. To report an error, write to us through the contact page.

Sources

  1. 1 Jean-Stanislas Denain and Chris Barber, “An FAQ on reinforcement learning environments”, Epoch AI Gradient Updates, Jan 12, 2026
  2. 2 SemiAnalysis, “RL Environments and RL for Science: Data Foundries and Multi-Agent Architectures”, Jan 6, 2026
  3. 3 TechCrunch, “Silicon Valley bets big on ‘environments’ to train AI agents”, Sep 16, 2025 (updated Sep 21, 2025)
  4. 4 Ege Erdil, Matthew Barnett and Tamay Besiroglu, “Cheap RL tasks will waste compute”, Mechanize, Aug 18, 2025
  5. 5 Prime Intellect, “Environments Hub: A Community Hub To Scale RL To Open AGI”, Aug 27, 2025
  6. 6 Lambert et al., “Tulu 3: Pushing Frontiers in Open Language Model Post-Training”, arXiv 2411.15124, Nov 22, 2024
  7. 7 DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning”, arXiv 2501.12948 v1, Jan 22, 2025
  8. 8 Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, arXiv 2310.06770, Oct 10, 2023
  9. 9 Xie et al., “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments”, arXiv 2404.07972, Apr 11, 2024
  10. 10 OpenAI, “Graders”, OpenAI API documentation, read Nov 18, 2026
  11. 11 Zhu et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks”, arXiv 2507.02825, Jul 3, 2025
  12. 12 Aleithan et al., “SWE-Bench+: Enhanced Coding Benchmark for LLMs”, arXiv 2410.06992, Oct 9, 2024
  13. 13 Skalse et al., “Defining and Characterizing Reward Hacking”, arXiv 2209.13085, Sep 27, 2022
  14. 14 Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale”, arXiv 2503.14476, Mar 18, 2025
  15. 15 Deshpande, Kannappan and Qian, “Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis”, ICML 2026, arXiv 2601.20103
  16. 16 “MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments”, Patronus AI, arXiv 2510.01353, Oct 1, 2025
  17. 17 “GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations”, Patronus AI, arXiv 2610.22708, Oct 15, 2026
  18. 18 “Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL”, Patronus AI, NeurIPS 2026, arXiv 2607.16204
  19. 19 “Adaptive Environments For LLM Agents Need to Jointly Scale Task, Scaffolding and Reward Design”, Patronus AI, Feb 18, 2026
  20. 20 Patronus AI, “Introducing Generative Simulators: Autonomously Scaling Environments for Agents”, Dec 17, 2025
  21. 21 Patronus AI, Domains and “Questions about buying environments”, read Nov 18, 2026
  22. 22 Patronus AI, “Announcing our $50M Series B to Simulate the Entire World’s Intelligence and Unveiling our First Digital World Model for AI Agent Training and Simulation”, Jun 25, 2026

Patronus AI · November 2026 snapshot · More resources