The number one way to justify a task's worth is to quantify a model's gain when the task is used in training.
When training using supervised fine-tuning (SFT) then reinforcement learning (RL), we achieved a performance jump from 35.6% to 47.8% on held-out GitHub computer use tasks and to 45.4% with RL alone. On held-out Unreal Engine development tasks, performance went from 64.3% to 71.5% with SFT then RL and to 73.1% with RL alone. Although the absolute gain within the GitHub tasks was greater, both task families fixed about one-fifth of the starting model's failures.
You'll note that RL on the tasks alone performed about as well as SFT then RL. This is aligned with research from the broader community, including DeepSWE and SWE-RL, which found that RL alone was a similar boon to coding agents. We see the same in our work with computer use and Unreal Engine development.
The sets of tasks in this early experimentation were small, with 46 held-out GitHub tasks and 54 held-out Unreal tasks. Given the gains are larger than the noise, the improvement over the starting model is real.
The gap between SFT then RL and RL alone is roughly 2 percentage points, which is within the noise, and we cannot say for sure which method is more effective.
As we scaled our task pipelines, it became apparent that we needed to build out our internal model post-training stack. This writeup is an exploration of that stack and initial findings we produced in training open-weight models through it.
The platform, from training to leaderboard
How can we track myriad, parallel experiments across researchers? How do we share findings across experiments efficiently? And most importantly, how can we reproduce findings at scale?
Our platform ran each stage of these experiments end to end.
Five stages loop to construct our training stack. Currently, we've engineered scripts that connect these steps. We will continue to automate these stages until a single training command can be run to generate comparable numbers on the other side.
First, we run a command that kicks off SFT or reinforcement learning. From there, checkpoints upload on their own, get merged into full models, and show up within a run's data store. A single click can turn a checkpoint into an OpenAI-compatible vLLM endpoint, one model per node, without manual cluster setup or provisioning. The evaluation platform then connects to this endpoint and runs our benchmark suite of held-out tasks under one shared preset. Finally, scores land on a shared leaderboard that compares the base and post-trained models under identical conditions, with the same preset, dataset hashes, and environments.
Although it's the least flashy of the five, the leaderboard is what matters most to our research team.
The only way we can guarantee proper evaluation and comparison at scale is by holding test tasks, attempts, and infrastructure failure rules constant across each model and test. We created a leaderboard that serves results and their rich context to enable anyone on our team to trust a number without rerunning experiments.
In pursuit of this goal, the platform snapshots the code, packs a deterministic source tarball, records the git commit and any uncommitted changes, uploads it to our system, and launches jobs through Slurm.
The snapshot enables comparison while making it easier to repeat runs. Repeating a run is as easy as cloning and restoring previous code and config. Cancellation and sweeps across up to 50 jobs are available on a simple web dashboard that links each run to its Weights & Biases page.
We test our platform like a live product. Unit and contract tests and rigorous CI are essential to enable us to build at scale while maintaining internal code quality.
Several of the tools we use in our training stack are standard and open source. We pack these tools as environments, such that setting them up on a node can be accomplished in seconds to minutes. For supervised fine-tuning, we used NVIDIA NeMo Automodel with LoRA on the language model and context parallelism for long trajectories. For reinforcement learning, we used NeMo-RL with GRPO. Rollouts are generated in vLLM and share GPUs with training.
The tasks themselves run outside this GPU cluster on our own environment service. Each attempt gets its own sandbox with the real application, an agent's tools, and a grader for the task. During reinforcement learning and evaluation, the agent picks an action (such as a click or a file edit), and the sandbox carries out each action and shows its result.
How we ran the experiments
Our first test involved two recipes: supervised fine-tuning then reinforcement learning, and reinforcement learning alone. We trained an open-source model, Qwen3.8-27B, which Alibaba has already post-trained, on two task families:
- Computer use tasks in the GitHub application. These used version control, worked in a browser with screenshots, and were associated with coding tasks.
- Tasks in Unreal Engine development, with code and editor work across five separate Unreal Engine projects.
Qwen3.8-27B was selected due to its relative balance of performance and size. It was performant enough that the gains were applicable justification for the buildout and small enough that we were not burning compute at an unnecessary scale.
Near copies of a task, like a reworded variant, are grouped together, and each whole group is placed entirely within or held out of the training set. This is done to prevent a model from merely memorizing a training task and thus excelling on a near copy of it in the held-out set.
Held-out tasks are spread across projects and difficulty levels and are limited to tasks that an expert model has solved at least once. We use Kimi-K3 as our expert in this case and additionally as the model that wrote the supervised fine-tuning data. This configuration mildly favors the SFT recipe.
Supervised fine-tuning trains on successful trajectories from Kimi-K3. We keep at most 4 trajectories per task and do not include any that are more than twice as long as the task's shortest solution. We do this so that the model learns efficient problem solving.
As for reinforcement learning, we use GRPO. The reward is 1 when a task's grader passes.
Calibrating task difficulty is often a challenging step in constructing reinforcement learning tasks. Models only learn when tasks are chosen so that some model attempts succeed and others fail. We aim to pick tasks tailored to a specific model's abilities to split this gap.
Within the GitHub task family, we selected tasks based on how often the expert model solved them. This resulted in 175 tasks for SFT then RL, and 139 for RL alone when dropping the tasks the first run solved every time.
As for Unreal tasks, we picked tasks by how often the SFT model solved them, which resulted in 53 tasks used for both recipes. Full training settings are in the appendix.
The evaluation step runs every held-out task on the model eight times to generate results. Success on these tasks is averaged per task and then across all tasks to produce our final output. A few percent of these attempts fail because the environment itself breaks.
These hiccups hit each model about equally, so including or excluding them changes no gain by more than a percentage point.
Results on held-out tasks
Our initial training jobs resulted in both recipes beating the starting model on both task families. The starting Qwen model solved 35.6% of our held-out GitHub tasks. SFT on just 41 trajectories gives a lift of 5.4 points, which is too small to separate from noise. RL then adds 6.8 points by step 40 for a total of 47.8%, which represents a 12.2-point jump over the starting model's performance. RL alone resulted in 45.4% solves at step 40 (+9.8). We continued training on GitHub tasks to step 60 and observed RL alone reaching 49.2% (+13.6).
As for the Unreal Engine task family, the starting model already solved 64.3% of held-out tasks, so there is less room for improvement. SFT then RL achieved 71.5%, a 7.2-point jump, and RL alone rose to 73.1%, an 8.8-point jump. The failure rate fell by roughly one-fifth with SFT then RL, and about a quarter with RL alone.
Overall, at step 40, success on held-out tasks rose between 7 and 12 points in both families. Most tasks got better and only a select few got worse.
Even the low end of each range is well above zero. These are fixed checkpoints chosen before evaluation and do not represent the best-scoring checkpoint.
Did the model cheat?
In the interest of verifying our environment's quality and findings' legitimacy, it was necessary for us to confirm the model was not reward hacking, and that passing runs were in fact attributable to real model gains.
We chose 50 passing runs from reported checkpoints, spread evenly across task families and weighted toward tasks that improved the most. After reviewing these runs by hand, we found none where grader code was read or an answer was copied, and that 48 of them were genuine solves. The two remaining runs were incorrectly flagged as complete when the GitHub grader passed them, even though they had run out of turns before finishing a required step.
After encountering this finding, we re-evaluated our runs and found that when counting every run that hit a turn limit as a failure, the gain across recipes and task families dropped by about a point at most, which was still well outside the noise.
This audit also found that the Unreal Engine sandbox left the grading harness readable to the model. We encountered a few runs that looked at this grader and three runs that attempted to download the original project files. Each was blocked by the closed network setup, and none of them resulted in a pass.
Inspecting a single trace
Although total gain is a good indication of training effectiveness, taking a closer look at individual traces from a run or comparing between runs is often necessary to understand where gains are coming from. We save detailed traces to our platform throughout our experiments, training, and inference. Analyzing specific examples such as the one below, where the starting model fails and the after-training model succeeds, can allow us to draw tighter links between the tasks we train on and the outcomes we achieve.
A specific example we pulled from our set of GitHub tasks involved retiring an old feature called PublishBatch. The code had a small counter that tallied every time someone used the feature. There were two places in the environment that said the counter had to stay active until the feature was removed. Someone prematurely opened a pull request that deleted the counter and the warning one release before it was due in v4.6. In the task, the model was asked to act as a code reviewer and leave a review on this specific pull request with two constraints:
- To cite one specific file: session.go
- To follow the project's review template
This template had fixed sections, including a Next Steps follow-up section. In this example, the counter was not present in session.go, which necessitated the model pulling the context from sources it was not explicitly tasked with checking. Both the starting model and the after-reinforcement-learning model saw the rules.
The figure compares the starting model with the same model after 60 steps of RL alone. The starting model called the early removal "a legitimate review concern," but it then dropped it because the counter was not explicitly listed in session.go. This resulted in the starting model saying that the PR was "fine to merge as-is." Our model after reinforcement learning was able to keep the concern across context. The after-RL model cited session.go but also flagged the early removal of the counter in the Next Steps section of the formatted output. We can see that RL taught the model to flag when a change breaks the project's own rules, inside the format the task asks for, rather than approving a request because the format seems to leave no room to rebut.
What we learned building the platform
We find that running these experiments ourselves is the best way to intuit what infrastructure is actually needed and to understand the nuances of the tools we build. These findings are as follows:
- Long runs need to restart themselves. Six Unreal RL runs were stopped by the same inference-server crash. After resuming the first by hand, we decided to automate the restart process to save researcher time and combat waking up to failed training runs. Later runs resumed from their last checkpoint on their own, with each recorded as a run linked to the one that came before it.
- A niche failure can be completely silent. When the environment service has an outage, reinforcement learning jobs sometimes don't crash themselves. Every attempt fails outright, rewards result in zero, and steps train nothing. One such run lost 8 of its 50 steps, and if we had not watched specifically for steps where every attempt fails, we would have missed it and perhaps attributed it to poor task quality or any other factor that prevented model gains.
- A single faulty node can look like a training bug. In one case, a node with broken DNS was failing jobs before they even started. Resubmitting jobs caused further failures, and fixing the node's DNS settings themselves was the only way to solve the issue.
- Long tasks require long context. Roughly one-third of Unreal training tasks require more than 150,000 tokens even in the shortest known solution, and these extend up to 376,000 tokens. Our nascent infrastructure was constrained to tasks that fit in 128,000 tokens, meaning a large minority of tasks in this set were not able to be used in these runs. Training on the rest requires infrastructure that serves, trains, and evaluates up to 256,000-token context windows or beyond.
What's next
Although these early results are promising, there is still much to build. Our platform works, but aspects are still manual. The RL worked, but it learned from blunt pass-or-fail signals. Our results have left us with open questions regarding longer runs and tasks. We are left with three areas of exploration: infrastructure, algorithms, and experimentation.
Infrastructure
Our infrastructure buildout involves three things. First, we are building a data flywheel. Every experiment, training, and inference job we run produces trajectories. We store and analyze these trajectories, and each failure can contribute to the next run's training data. As our experiments scale, so do our learnings and the training data for each subsequent run.
Second, we are building a unified evaluation harness. The high performance of today's frontier models necessitates complex agentic benchmarks each with their own setup to communicate with a model. If we successfully build a harness as one unified layer between benchmarks and a model, adding a new benchmark to our stack will not require as much setup cost.
Third, we are working on online evaluation. All of the runs in this post were done offline. By moving them online, we will enable engineers inside and outside of our team to experiment with checkpoints directly and catch things that benchmarks might miss.
Algorithms
We are exploring training algorithms. Today's rewards are all or nothing, black or white, one or zero. This lack of nuance limits what a model can learn through RL. We are exploring three research directions in the vein of adding nuance to the scoring of RL tasks.
The first is partial credit. Often in life, we are rewarded for partial progress. Within very difficult tasks, where complete solves are rare, even the most performant models may never achieve complete success. Allowing them partial credit could enable gains that would otherwise have gone unrealized.
Second is allocating credit across steps in new ways. The tasks we work with are often on the order of tens of turns. If there is one singular reward value at the end of these turns, necessarily this score is applied to all turns equally. We are exploring what it means to reward each individual action.
Third, we are exploring adaptive task choice. All of our experiments outlined here involve one single task set chosen up front. Imagine a task in this set was thoroughly solved by the model and was reliably passed on every single attempt after step 10. This task's inclusion between steps 10 and 40 is unnecessary. To produce further results within the same constraints on compute or infrastructure, we are exploring a system to dynamically choose tasks after previous tasks have been thoroughly solved.
Experimentation
Further experimentation is required to expand these results. First, we are continuing to experiment with longer RL runs. GitHub RL performance was still rising at step 60 with no indication that gains were slowing. We assume that training longer will result in further gains, and continuing these longer runs can demonstrate where our curve flattens or plateaus. Second, we will experiment with harder and longer-horizon Unreal tasks at a context length of 256,000 tokens. We will run them with the same split, SFT and evaluation steps and compare results directly with those outlined in this writeup. Finally, we will explore running rollouts asynchronously across fewer nodes to produce cheaper results. We will only build out this new direction once we verify that the quality of output is replicated on this new setup.
We work on these items every day. We are hiring researchers and engineers to build them with us. patronus.ai/join
Appendix, training details
