Alignment
Follow instructions and know when to push back
A model can do what it was told and still miss what was meant, or state something false with confidence. We study these failures and build training environments around them.
RL environments for alignment, built from how models misbehave
RL environments for alignment teach a model to do what was meant, not only what was measured. They are AI alignment datasets built as training environments. A model works through a realistic task, and the reward looks at how it got there as well as the result.
Rewards that check only the outcome can teach the wrong lesson. In a coding run, a model that deletes a failing test can earn the same reward as one that fixes the bug. Our TRACE benchmark cataloged 54 kinds of reward hack in code environments. The best model tested caught 63% of them with runs side by side, and 45% when it saw one run at a time.
We study failures of behavior, not skill. In a behavioral failure the work may pass, but the process is wrong. The model claims a check it never ran, or agrees with a user it should correct. Hallucination is part of this. A benchmark our team contributed to, HalluWorld, measures a false claim against a fully specified reference world, so it can be labeled without guesswork.
Environments also have to look real. A model that notices it is being tested can behave differently. We build worlds with the history and detail a working engineer would expect to find.
Published research
TRACE
54 kinds of reward exploit in code RL, and how well models catch them.
HalluWorld
A controlled benchmark that measures hallucination against reference world models.
Unified Hallucination Framework
One definition of hallucination: inaccurate world modeling.
Questions about AI alignment datasets
What is an AI alignment dataset?
It is a set of tasks that rewards a model for behaving the way people intend. Many are pairs of preferred and rejected answers. Ours are environments, where a model does multi-step work and misbehavior can show up along the way.
What is reward hacking?
Reward hacking is when a model raises its reward without doing the task, for example by editing a test instead of the code. It is a known problem in reinforcement learning on code. TRACE measures how well models catch it.
How do you tell a behavioral failure from a capability gap?
We compare what the model did with what a reasonable professional would have done in its place. If the model had what it needed and still didn’t do the work, the failure is behavioral, not a gap in skill.
Can models check other models for alignment failures?
Partly. In TRACE, models caught reward hacks more often when they could compare runs than when they judged one run alone. Whether model reviewers can reliably catch subtler failures, such as sycophancy, is an open question we are studying.
Do you measure hallucination?
Yes. HalluWorld measures hallucination in gridworlds, chess and terminal tasks where the true state of the world is known, so labels are generated automatically. A companion paper argues that hallucination is best defined as inaccurate world modeling.