Pass@k and RL difficulty calibration

Pass@1 is the chance a single attempt succeeds. Pass@8 is the chance that at least one of eight does. The metric comes from code generation, where Chen et al. (2021) measured it by sampling n attempts per problem, with n at least k, counting the c that pass, and estimating pass@k as 1 minus C(n − c, k) divided by C(n, k). That estimator is unbiased, which matters when n is small.

Why difficulty matters in RL

RL learns from differences between attempts. A problem the model never solves gives no signal, and neither does one it always solves. In GRPO, the algorithm our runs use, each attempt is scored against the other attempts in its group (Shao et al., 2024), so when every attempt in a group gets the same reward, the reward gives that group no learning signal. We have seen one end of this in our own training, where an environment outage made every attempt fail and those steps trained nothing. The other end is a problem the model passes on every attempt after step 10, which is unnecessary from step 10 to step 40 (post-training stack).

So the useful problems for a model are the ones it sometimes solves, and models only learn when some attempts succeed and others fail (same source). Pass@k across a few attempts shows where each problem sits for that model.

How we calibrate difficulty

We set difficulty against the models a team is training and retune problems as new models arrive (/domains/coding). The same problem can be hard for one model and easy for the next, so a difficulty label without a model attached says little.

We also prefer difficulty that comes from the work. Our coding problems are hard because they are underspecified and spread across tools (same source). In MEMTRACK, the facts an engineer needed were spread across Slack, Linear and Git, and the best model tested reached 60% correctness.

Difficulty can also move with the agent. Our Generative Simulators work filters generated problems by the current agent's capabilities and the difficulty required, and problem generation, tooling and reward can each be made harder on their own (post).

How we measure it

In our post-training evaluation, every held-out problem runs eight times. Success is averaged per problem and then across problems (post-training stack). Held-out problems are spread across projects and difficulty levels, and limited to problems an expert model has solved at least once, so every one is known to be solvable (same source). Near copies of a problem, such as a reworded variant, are kept together on one side of the split, so a model can't pass a held-out problem by memorizing its twin.

Averages over small sets are noisy. With 46 held-out GitHub problems and 54 held-out Unreal problems, a gap of about 2 percentage points between two training recipes was within the noise, and we did not call a winner (same source).

A checklist for calibrating RL problems

1. Measure pass@k on the model you will train, not on a different or older one.
2. Use enough attempts per problem for the estimate to settle. We use eight.
3. Drop or rework problems at pass@k of zero or one for that model. Confirm unsolved ones are solvable with a reference solution.
4. Recheck when the model changes, and during training, since problems the model has mastered stop teaching it. We are exploring choosing problems adaptively as training goes (post-training stack).
5. Separate environment failures from model failures. A few percent of our attempts fail because the environment itself breaks (same source).