Glossary

RL environment and post-training glossary

Short definitions of the terms used in building RL environments and post-training AI agents, each tied to the Patronus AI paper, guide, domain page or post it comes from.

Reviewed by Evan JolleyUpdated 61 terms

Index

Theme 01

Environments

16 terms

Back to index ↑

Reinforcement learning

Also called RL, RL in AI, reinforcement learning in AI

Reinforcement learning (RL) is a way of training AI in which an agent learns by trial and error, acting in an environment and adjusting to the rewards its actions earn.

In Patronus's work
Patronus builds the environments and rewards that RL for language models trains on, across coding, computer use, knowledge work, science, alignment and more. Its own post-training runs train a model with RL on problems in the GitHub application, worked in a browser, and in Unreal Engine projects, then test it on held-out problems (post-training stack).
Example
In Patronus's RL runs the reward is 1 when a problem's grader passes and 0 otherwise (post-training stack).

RL environment

Also called reinforcement learning environment, training environment

An RL environment is a sandbox with the tools an agent needs, a problem to solve and a reward that scores the result.

RL gym

Also called training gym, gym

RL gym is another name for an RL environment, after the "gym" software interfaces that many RL environments follow.

Read more
RL environments guide, and the RL environment entry

Problem

Also called RL problem

A problem is one unit of work inside an RL environment, a goal the agent must reach from a starting state, with a way to score whether it did.

In Patronus's work
Patronus coding problems start from a real repository and don't say where to look, so the agent has to explore the code to decide what to do. Experts review every problem before it is used (/domains/coding).
Example
A coding problem can range from a small bug fix to a feature that touches the database, the API and the frontend (/domains/coding).
Read more
/domains

Task

Also called RL task, environment task, task instance

A task is one unit of work an agent is given in an RL environment or benchmark, the same thing Patronus calls a problem in this glossary.

In Patronus's work
Live Patronus pages use both words for the same unit. The coding domain page says experts review every task before it is used (/domains/coding), and this glossary says problem throughout. See Problem for the full entry.
Example
Patronus's post-training evaluation runs every held-out task on the model eight times, and each run counts as one attempt at that problem (post-training stack).

Seed state

Also called initial state, starting state, deterministic reset

A seed state is the exact starting condition of an RL environment, the files, data and app state an agent finds at the start of every attempt.

In Patronus's work
Patronus coding environments start from a real repository with its file tree, commit and pull request history, tests, docs and run commands (coding FAQ). In its own training, each attempt gets its own sandbox, so attempts don't share state (post-training stack).
Example
Patronus runs every held-out problem eight times, and each attempt gets its own sandbox, so no attempt inherits another's changes (post-training stack).

Sandbox

Also called execution sandbox, isolated environment

A sandbox is an isolated computing space where an agent can run commands and change files or apps without affecting anything outside it.

In Patronus's work
Patronus coding RL environments are sandboxes with a codebase, tools, a problem and a reward (coding FAQ). In Patronus's training stack the problems run on a separate environment service, and each attempt gets its own sandbox with the application, the agent's tools and a grader (post-training stack).
Example
The agent picks an action, such as a click or a file edit, and the sandbox carries it out and shows the result (post-training stack).

Simulated app

Also called simulated apps, app simulation, app clone, mock website

A simulated app is a working copy of a piece of software, built so an AI agent can operate it inside an RL environment and have its actions checked.

In Patronus's work
Simulated apps and mock websites are among the parts of an RL environment that Patronus sells, on their own or as a whole environment (/domains buying FAQ). The same environment can serve agents that work from screenshots and browser agents that read the page structure (/domains/computer-use).
Example
Experts complete each computer use problem by hand to confirm it can be solved before it is used (/domains/computer-use).
Read the full entry →

Multi-app environment

Also called multi-platform environment, cross-app environment, enterprise environment

A multi-app environment is an RL environment that spans several connected tools, so an agent has to gather and act on information across them.

In Patronus's work
In MEMTRACK (NeurIPS 2025 Workshop), the facts an engineer needed were spread across Slack, Linear and Git, with noisy and conflicting information, and the best model tested reached 60% correctness. Knowledge work environments hold the documents, tools and messages of a job (/domains/knowledge-work).
Example
Patronus finance problems run in spreadsheets, terminal-style tools and document apps, and many span several apps and long sequences of steps where the state changes along the way (/domains/knowledge-work).
Read more
MEMTRACK

Golden trajectory

Also called reference solution, worked solution, expert trajectory, gold trajectory

A golden trajectory is a known-good, step-by-step solution to a problem, used to prove the problem can be solved and to test the grader.

In Patronus's work
Patronus tests each grader against worked solutions, so a low score means the model was wrong (/domains/knowledge-work). For computer use, it records experts as they work and turns the recordings into trajectories, as in FigmaTrace.
Example
In Patronus's post-training runs, held-out problems are limited to ones an expert model has solved at least once (post-training stack).

Generative simulator

Also called generative simulation, generative environment, autonomously scaling environment

A generative simulator is an adaptive RL environment that generates its problems, world dynamics and reward functions together, so difficulty can rise as the agent improves.

In Patronus's work
Patronus introduced Generative Simulators in a December 2025 technical report. Problem generation, tooling and reward can each be made harder on their own or together, which helps raise difficulty where a model struggles (post).
Example
A curriculum filter picks the problems the agent runs from the generated set, and a reward function scores its trajectories into a report that feeds back into the difficulty scale (post).

Language model as environment

Also called text world model, simulated environment, LM-simulated environment

A language model as environment is a model trained to predict how an environment responds to an agent's actions, so it can stand in for that environment during training.

In Patronus's work
World Modeling for Agentic RL (NeurIPS 2026) curates 239,403 state-action trajectories and finds masked diffusion language models beat autoregressive language models over 4x their size on coherence, groundedness and rollout diversity. GlobeBench found the best model succeeded 49.57% of the time.
Example
In GlobeBench, success fell to an average of 5.1% across 20 models when an agent's change had to spread to dependent objects.

Off-the-shelf vs custom RL environments

Also called buy vs build RL environments, ready-made RL environments, custom RL environments

Off-the-shelf RL environments are ready-made and sold as they are, while custom environments are built to order for one team's model, domain and difficulty.

In Patronus's work
Patronus sells both. It has ready-made environments, but most of its work is custom, and buyers can also buy single parts such as simulated apps (/domains buying FAQ). Custom builds usually start with samples.
Example
A team with its own environment framework might buy only the mock websites and the data behind them (/domains buying FAQ).

Training harness

Also called RL harness, training framework

A training harness is the software that connects a model to its environments during training, sending actions, collecting results and rewards, and feeding them to the training algorithm.

In Patronus's work
Patronus confirms during scoping how its problems can run on a buyer's own harness (/domains buying FAQ). Its own stack runs RL on GPUs and the problems on a separate environment service, and it is building one evaluation harness between benchmarks and models (post-training stack).
Example
When the environment service has an outage, RL jobs may not crash, every attempt scores zero, and the step trains nothing. One run lost 8 of its 50 steps this way (post-training stack).

Long-horizon problem

Also called long-horizon task, multi-step task, long-horizon agent task

A long-horizon problem is one that takes an agent many steps or turns to solve, so early mistakes surface late and the reward arrives only at the end.

In Patronus's work
About a third of Patronus's Unreal Engine training problems need more than 150,000 tokens even in the shortest known solution, up to 376,000, too long for its first training runs (post-training stack). SpeedrunBench found agents near human records on simple platformers but behind on longer, more complex games.
Example
A computer use problem doesn't say which buttons to press, so the agent has to find its way and keep its place over dozens of steps (/domains/computer-use).
Read more
MEMTRACK

Theme 02

Rewards and grading

14 terms

Back to index ↑

Verifier

Also called reward verifier, verifiable reward, rule-based reward

A verifier is a check, usually code, that decides whether an agent's answer or final state is correct, giving an RL environment a reward it can trust.

In Patronus's work
In coding, tests are the starting point, and Patronus adds the checks a reviewer would make where a problem needs more (/domains/coding). In finance, where an answer sits in the source records, code checks it against those records (/domains/knowledge-work).
Example
A model that deletes a failing test can earn the same reward as one that fixes the bug, if the verifier only checks that tests pass (/domains/alignment).
Read the full entry →

Grader

Also called grading function, scorer, reward checker

A grader is the part of an RL environment that scores an agent's attempt, by code, by a rubric applied by a model, or both.

In Patronus's work
In Patronus's training runs each sandbox includes a grader for its problem, and the reward is 1 when the grader passes (post-training stack). For finance, practitioners who have done the job write each problem and its checks, and separate experts review them (/domains/knowledge-work).
Example
A credit memo has no single right answer, so practitioners write the grading criteria (/domains/knowledge-work).

Grading accuracy

Also called grader validation, grader accuracy, reward accuracy

Grading accuracy is how often an environment's grader gives the score a qualified expert would give the same attempt.

In Patronus's work
Patronus tests each grader against worked solutions, so a low score means the model was wrong (/domains/knowledge-work). Every environment also gets manual review by domain experts and automated checks before delivery (/domains buying FAQ).
Example
Patronus checked that passing runs in its post-training experiments came from model gains and not reward hacking (post-training stack).

Rubric

Also called rubric-based reward, grading criteria, rubric grading

A rubric is a written list of criteria for judging an answer that has no single right form, used to grade open-ended work in an RL environment.

In Patronus's work
Where work has no one right answer, such as a credit memo, practitioners from the job write the grading criteria (/domains/knowledge-work). In science, Patronus leans toward realistic prompts and does the extra work to grade them (/domains/science).
Example
A finance problem ends in a deliverable a practitioner would accept, like a valuation or an audit finding (/domains/knowledge-work).

Rule-based vs model-based rewards

Also called rule-based verifiers vs LLM as a judge, programmatic vs model rewards

Rule-based rewards come from code that checks a result exactly, while model-based rewards come from a model that judges the result, often against a rubric.

In Patronus's work
Patronus uses code checks where a right answer exists and practitioner-written criteria where it doesn't (/domains/knowledge-work). Model judges have known biases. Contrastive Decoding for LLM Judges (Findings of ACL 2026) shows judge scores are highly sensitive to the score range given, and reduces that bias.
Example
A balance in a ledger can be checked by code, while a credit memo needs written criteria (/domains/knowledge-work).

Process reward vs outcome reward

Also called process supervision, outcome supervision, step-level reward

An outcome reward scores only the final result, while a process reward also scores how the agent got there, step by step.

In Patronus's work
Patronus builds rewards that look at how a model got there as well as the result, because outcome-only rewards can teach the wrong lesson (/domains/alignment). Its post-training team is exploring partial credit and rewarding individual steps, since one reward at the end of tens of turns is applied to every turn equally (post-training stack).
Example
A model can claim a check it never ran. The outcome may pass while the process is wrong (/domains/alignment).

Reward

Also called reward signal

A reward is the score an RL environment gives an agent for an action or an attempt, the signal training uses to make better behavior more likely.

Reward function

Also called reward design

A reward function is the rule that turns what an agent did into a reward, written by hand, generated, or learned.

Reward model

Also called learned reward model, preference model

A reward model is a learned model that predicts how good a response is, used as the reward when no exact check exists.

Sparse reward

Also called binary reward, pass or fail reward

A sparse reward is one an agent gets rarely, often only at the end of a long attempt, which makes learning which steps helped harder.

Read more
RLVR guide

Verification gap

Also called verifier coverage gap

A verification gap is the space between what a verifier checks and what success really requires, where incomplete checks accept wrong answers.

Read more
RLVR guide

RLVR

Also called reinforcement learning with verifiable rewards, RL with verifiable rewards

RLVR (reinforcement learning with verifiable rewards) is RL in which the reward comes from an automatic check of correctness, such as a test or an exact answer.

Read more
RLVR guide, and verifier

Reward hacking

Also called specification gaming, reward exploit, reward gaming

Reward hacking is when a model raises its reward without doing what the problem asked, by exploiting gaps in how the reward is checked.

In Patronus's work
Patronus's TRACE benchmark (ICML 2026) catalogs 54 kinds of reward exploit in code environments over 517 trajectories. The best model tested caught 63% of hacks when it could compare runs, and 45% when it judged one run alone.
Example
A model edits a test instead of the code, so the test passes and the bug stays (/domains/alignment).

Theme 03

Training methods

13 terms

Back to index ↑

Post-training data

Also called AI training data for post-training, agent training data

Post-training data is what a model learns from after pretraining, such as expert examples, preference pairs, and environments with rewards.

In Patronus's work
Patronus builds post-training data as environments with problems and rewards, plus expert trajectories. A code dataset is static text, while agent training data captures the work itself, from the files an agent opened to the change it made (/domains/coding). The same environment can serve training and evaluation.
Example
FigmaTrace turns over 200 hours of expert screen recordings into 3,469 design trajectories.

Expert data

Also called human expert data, expert demonstrations, expert-written data

Expert data is training data written, recorded or checked by people who do the job, such as engineers, analysts or scientists.

In Patronus's work
Patronus builds each knowledge work environment with practitioners from the job, and experts who didn't build a problem review it before it ships (/domains/knowledge-work). Patronus hires experts for this through /join.
Example
In FigmaTrace, a model trained on expert Figma recordings improved on GUI agent benchmarks outside design (/domains/computer-use, FigmaTrace).
Read more
/join

RLHF

Also called reinforcement learning from human feedback

RLHF (reinforcement learning from human feedback) is RL in which the reward comes from human judgments, usually through a reward model trained on them.

GRPO and PPO

Also called group relative policy optimization, proximal policy optimization

PPO and GRPO are RL algorithms commonly used to train language models. GRPO scores each attempt against a group of attempts at the same prompt.

Read more
Online RL guide, and Patronus's post-training stack, which uses GRPO

Online RL

Also called online reinforcement learning

Online RL is RL in which the model generates fresh attempts during training and learns from their rewards, in a live loop.

Model-based RL

Also called model-based reinforcement learning

Model-based RL trains an agent with a learned model of the environment's dynamics, so it can imagine the outcomes of actions before taking them.

RL training loop

Also called RL training, RL loop

The RL training loop repeats three steps, the model attempts problems, a reward scores the attempts, and the model's weights update toward what scored well.

Policy

Also called agent policy

A policy is the agent's rule for choosing actions. For a language model agent, the policy is the model itself.

Rollout

Also called episode, attempt

A rollout is one full run of an agent in an environment, from the starting state to the end, which produces a trajectory and a reward.

Transfer across domains

Also called cross-domain transfer, generalization

Transfer across domains is when training on one kind of problem improves a model on another kind it was not trained on.

In Patronus's work
There is public evidence that models that improve at coding also improve at computer use and tool use, but transfer to other domains is less clear, so Patronus doesn't assume it (/domains/coding).
Example
In FigmaTrace, a 27B open model trained on expert design trajectories improved on five of six GUI agent benchmarks outside design.

Theme 04

Evaluation and failure modes

9 terms

Back to index ↑

Pass@k

Also called pass at k, pass@1, difficulty calibration, RL task difficulty

Pass@k is the chance that at least one of k attempts at a problem succeeds, used to measure how hard a problem is for a given model.

In Patronus's work
Patronus sets difficulty against the models a team is training and retunes problems as new models arrive (/domains/coding). Its post-training evaluation runs every held-out problem eight times and averages success per problem, then across problems (post-training stack).
Example
A problem the model passes on every attempt after step 10 adds nothing from step 10 to 40, which is why Patronus is exploring adaptive problem choice (post-training stack).
Read the full entry →

Benchmark vs environment

Also called benchmark vs RL environment, evaluation vs training environment

A benchmark is a fixed set of problems for measuring models, while an RL environment lets a model act, earn rewards and train, and can also evaluate.

In Patronus's work
The same environment can serve training and evaluation (/domains/coding). FinanceBench, Patronus's open-book benchmark of 10,231 questions about public companies, is used by post-training teams as an environment (/domains/knowledge-work).
Example
Static benchmarks are susceptible to reward hacking, contamination, leakage and saturation, which is why Patronus works on environments that generate new problems (Generative Simulators post).
Read more
/research

RL evaluation

Also called reinforcement learning evaluation, evaluating RL models

RL evaluation measures how a trained agent performs on held-out problems, and feeds what it finds back into the data, graders and rewards.

Trajectory

Also called trace, agent trace, agent trajectory

A trajectory is the full record of an agent's attempt, every observation, thought, tool call and action from start to finish.

In Patronus's work
Trajectories are both training data and evidence. TRAIL is 148 human-annotated agent traces with a taxonomy of error types, and FigmaTrace converts expert recordings into 3,469 design trajectories. Patronus stores and analyzes every trajectory its experiments produce (post-training stack).
Example
In FigmaTrace, splitting long recordings by design phase worked better than cutting them to fit a context length (/domains/computer-use).
Read more
TRAIL

Error localization

Also called trace debugging, finding where an agent went wrong, agent error analysis

Error localization is finding the step in an agent's trajectory where it went wrong, and often naming the kind of error made there.

In Patronus's work
In TRAIL, the best model tested scored 11% at trace debugging, and the authors find modern long-context models do poorly at it.
Example
TRAIL's 148 hand-annotated traces, from software engineering and information retrieval agents, carry 841 labeled errors, each tied to the step where it happened (TRAIL).
Read more
TRAIL

Hallucination

Also called LLM hallucination, confabulation

A hallucination is an observable claim from a model that is false with respect to the world it is meant to describe, such as a source or a knowledge base.

In Patronus's work
The Unified Hallucination Framework (ICML 2026) defines hallucination as inaccurate world modeling observable to the user. HalluWorld (NeurIPS 2026) measures it in worlds whose true state is known, and finds multi-step state tracking still hard.
Example
Stating a fact that contradicts a knowledge base, or writing a summary that contradicts its source (Unified Hallucination Framework).

Behavioral failure

Also called behavior failure, misbehavior

A behavioral failure is when a model had what it needed to do the job and still didn't do it as intended, as opposed to a gap in skill.

In Patronus's work
Patronus tells a behavioral failure from a gap in skill by comparing what a model did with what a reasonable professional would have done in its place (/domains/alignment).
Example
The work may pass while the process is wrong, as when a model claims a check it never ran, or agrees with a user it should correct (/domains/alignment).

Sycophancy

Also called sycophantic behavior

Sycophancy is when a model tells users what they want to hear, agreeing with them or going along with a request, instead of correcting or refusing.

In Patronus's work
Whether model reviewers can reliably catch subtler failures such as sycophancy is an open question Patronus is studying (/domains/alignment).
Example
A model agrees with a user it should correct (/domains/alignment).

State tracking

Also called world state tracking, memory of state

State tracking is a model's ability to keep an accurate picture of how an environment has changed over many steps, including effects it didn't see directly.

In Patronus's work
HalluWorld finds multi-step state tracking and causal forward simulation remain hard for frontier models. GlobeBench (NeurIPS 2026 Workshop) finds accuracy drops as an action affects more of the environment.
Example
In GlobeBench, 631 of 1,632 problems remain unsolved by any of the 20 models tested.
Read more
GlobeBench

Theme 05

Agents and domains

9 terms

Back to index ↑

AI agent

Also called agent, LLM agent

An AI agent is a model that takes actions toward a goal over several steps, using tools, memory and the results of earlier steps.

Computer use agent

Also called CUA, computer-using agent, GUI agent

A computer use agent is an AI agent that operates software through its interface the way a person does, by reading the screen, clicking and typing.

Browser agent

Also called web agent

A browser agent is a computer use agent that works in a web browser, from screenshots, from the page structure, or both.

Coding agent

Also called SWE agent, software engineering agent

A coding agent is an AI agent that reads a codebase, runs commands and submits changes to complete a software engineering problem.

Knowledge work agent

Also called enterprise agent, professional agent

A knowledge work agent is an AI agent judged on a professional deliverable, such as a memo, a decision or a reconciled account.

Agentic memory

Also called agent memory, long-term memory for agents

Agentic memory is how an AI agent stores, recalls and uses information across many steps and interactions.

Tool use

Also called function calling, tool calling

Tool use is an AI agent calling functions, such as a search, a code runner or an API, to act on its environment.

MCP

Also called Model Context Protocol

MCP (Model Context Protocol) is an open standard for connecting AI agents to external tools and data in one consistent way.