ICML 2026TRACE
54 kinds of reward exploit in code RL, and how well models catch them.
Post-training data, measured in our own training runs
Reviewed by Zhe Li, Member of Technical Staff, Research
Post-training data is what a model learns from after pre-training. Some of it is examples to imitate, some is pairs of answers with the better one marked, and some is an environment where the model practices a task and earns a reward. Patronus AI builds RL environments, reviews agent transcripts in pairs, and measures what our tasks are worth by training open models on them.
- 7 to 12 points
- Gain in success on held-out tasks, in both task families we trained on
- About 1 in 5
- Starting-model failures fixed by training, in each family
What is post-training data?
Post-training data is what a model learns from after pre-training, to pick up a skill or a behavior with far less data than pre-training used. The data has to fit the method that trains on it. Our guide to LLM post-training covers the methods in depth.
| Data | What it is | What trains on it | In our work |
|---|---|---|---|
| Demonstrations and trajectories | Worked examples of a task done well, from a single ideal answer to a long trace of an agent’s steps. | Supervised fine-tuning (SFT), which trains the model to imitate the examples. | Successful attempts on our tasks can become SFT data. In our runs it came from an open model, Kimi-K3, kept only when the task’s grader passed. FigmaTrace, our research dataset of 3,469 human Figma design trajectories, is public. |
| Preference data | Two answers to the same prompt, with the better one marked. | A reward model trained on the comparisons, or direct preference optimization (DPO), which needs no reward model. | Expert pairwise reviews of agent transcripts, comparing two runs of the same task and marking the better one. |
| RL environments | A sandbox with the tools an agent needs, a task and a reward that scores the result. | Reinforcement learning. In our runs, GRPO with a reward of 1 when the task’s grader passes, a verifiable reward. | What we build. RL environments with tasks, graders and simulated apps for alignment, coding, knowledge work, science, computer use and more. |
How much does post-training data improve a model?
In our runs, training on our tasks raised success on held-out tasks by 7 to 12 points. The clearest test of a task is what a model gains from training on it, so we trained an open 27B model on two of our task families, computer use in GitHub and game engine development, and scored it on held-out tasks it never saw in training.
At step 40, on GitHub tasks success went from 35.6% to 47.8% with SFT then RL. On game engine tasks it went from 64.3% to 73.1% with RL alone. Each family fixed about one-fifth of the starting model’s failures.
RL alone did about as well as SFT then RL. The gap between them was about 2 points, which is within the noise, and that matches published research on coding agents. The sets were small, 46 held-out GitHub tasks and 54 game engine tasks, and the gains are larger than the noise. We chose the checkpoints before evaluation.
How do you measure post-training data quality?
Train a model on the data and test it on held-out tasks. These five controls keep our numbers honest.
Near copies stay together
A reworded variant of a task is grouped with the original, and each group goes wholly into training or wholly into the held-out set. A model can’t pass a held-out task by remembering its twin.
Eight runs per task
Every held-out task runs eight times. Success is averaged per task, then across tasks.
One shared leaderboard
The starting model and every trained model are scored on the same tasks, with the same settings, dataset hashes and environments.
Passes checked by hand
We reviewed 50 passing runs by hand. None read grader code or copied an answer, and 48 were genuine solves. The other two ran out of turns before a required step and the grader passed them anyway. Counting every run that hit the turn limit as a failure moved the gains by a point at most.
Silent failures caught
When the environment service goes down, an RL run can keep going while every attempt fails. One run lost 8 of its 50 steps this way. We watch for steps where every attempt fails, so an outage is never mistaken for weak tasks.
Post-training data by domain
Every environment is reviewed by domain experts and checked automatically before delivery.
- AlignmentFollow instructions and know when to push back.
- CodingShip a feature in code no one documented.
- Knowledge WorkMake one decision from sources that disagree.
- ScienceIdentify and explore the untested question.
- Computer UseGet work done in software built for people.
- And more. See every domain →
From delivery to your training stack
Most of our work is custom. We build the tasks in each environment and calibrate them to what you need, and we usually send samples first.
Ownership of the data from a custom build is set in each agreement, and we confirm during scoping how tasks run on your own training harness. The other buying questions are answered on our domains page.
For the engineering side of post-training at scale, read how we got reinforcement learning running on a 744B open model quantized to 4 bits.
Published research
ICML 2026
NeurIPS 2026World Modeling for Agentic RL
A language model that stands in for an environment while an agent trains.
Generative Simulators
Environments that generate their own tasks, world dynamics and rewards.
More research
- FigmaTrace2026 · arXiv 2608.21460
Questions about post-training data
What is post-training data?
Post-training data is the data a model trains on after pre-training to learn a skill or a behavior. It includes demonstrations and trajectories for supervised fine-tuning, preference pairs for reward models or DPO, and RL environments where a model practices a task and earns a reward.
What post-training data does Patronus AI build?
RL environments, with the tasks, graders and simulated apps a model practices in. We also do expert pairwise reviews of agent transcripts, marking the better of two runs. In our own runs, successful attempts on those tasks by an open model also served as SFT data. Details are in our post-training write-up.
How is SFT data different from preference data and RL environments?
SFT data shows the model a good answer to copy. Preference data shows two answers and marks the better one. An RL environment gives the model a task and scores what it does, so the model learns from its own attempts.
Does RL alone work as well as SFT then RL?
In our runs it did. On held-out tasks, RL alone came within about 2 points of SFT then RL, which is within the noise. SFT on 41 trajectories gave a lift too small to separate from noise. Details are in our post-training write-up.
How do you measure the quality of post-training data?
Train a model on it and test the model on held-out tasks. We keep near copies of a task out of the held-out set, run each held-out task eight times, score every model on one shared leaderboard and check passing runs by hand for reward hacking.
What should a post-training data provider deliver?
Tasks a model can learn from, with graders that hold up. A model learns only when some attempts succeed and others fail, so tasks should be calibrated to the model being trained. Every Patronus AI environment is reviewed by domain experts and checked automatically before delivery.
Need post-training data for your model?
Talk to us