No items found.

Alignment

Follow instructions and know when to push back

A model can do what it was told and still miss what was meant, or state something false with confidence. We study these failures and build training environments around them.

Joshua tree at sunset, Queen Valley, California. Photo: NPS, Joshua Tree National Park (public domain)Line drawing of Joshua tree at sunset, Queen Valley, California

RL environments for alignment, built from how models misbehave

RL environments for alignment teach a model to do what was meant, not only what was measured. They are AI alignment datasets built as training environments. A model works through a realistic task, and the reward looks at how it got there as well as the result.

Rewards that check only the outcome can teach the wrong lesson. In a coding run, a model that deletes a failing test can earn the same reward as one that fixes the bug. Our TRACE benchmark cataloged 54 kinds of reward hack in code environments. The best model tested caught 63% of them with runs side by side, and 45% when it saw one run at a time.

We study failures of behavior, not skill. In a behavioral failure the work may pass, but the process is wrong. The model claims a check it never ran, or agrees with a user it should correct. Hallucination is part of this. A benchmark our team contributed to, HalluWorld, measures a false claim against a fully specified reference world, so it can be labeled without guesswork.

Environments also have to look real. A model that notices it is being tested can behave differently. We build worlds with the history and detail a working engineer would expect to find.

Published research

“Alluvial Fan”, Taklimakan Desert, China“Alluvial Fan”, Taklimakan Desert, China
ICML 2026
meta

Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis

January 27, 2026
ICML 2026
Main
trace
ICML 2026
2026
“Serene Expressions”, off the Louisiana coast“Serene Expressions”, off the Louisiana coast
NeurIPS 2026
meta

HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models

May 19, 2026
NeurIPS 2026
Main
halluworld
NeurIPS 2026
2026
“Meighen Island”, Canadian Arctic“Meighen Island”, Canadian Arctic
ICML 2026
meta

Position: We Need A Unified Definition of Hallucination (It’s The World Model, Stupid!)

December 25, 2025
ICML 2026
Main
unified-hallucination-framework
ICML 2026
2025

Read all our research →

Questions about AI alignment datasets

What is an AI alignment dataset?

It is a set of tasks that rewards a model for behaving the way people intend. Many are pairs of preferred and rejected answers. Ours are environments, where a model does multi-step work and misbehavior can show up along the way.

What is reward hacking in reinforcement learning?

Reward hacking in reinforcement learning is when a model raises its reward without doing the task, for example by editing a test instead of the code. It is a known problem in reinforcement learning on code. TRACE measures how well models catch it.

How do you tell a behavioral failure from a capability gap?

We compare what the model did with what a reasonable professional would have done in its place. If the model had what it needed and still didn’t do the work, the failure is behavioral, not a gap in skill.

Can AI models detect reward hacking by other models?

Partly. In TRACE, models caught reward hacks more often when they could compare runs than when they judged one run alone. Whether model reviewers can reliably catch subtler failures, such as sycophancy, is an open question we are studying.

How do you benchmark LLM hallucination?

With HalluWorld, which measures hallucination in gridworlds, chess and terminal tasks where the true state of the world is known, so labels are generated automatically. A companion paper, Position: We Need A Unified Definition of Hallucination (It’s The World Model, Stupid!), argues that hallucination is best defined as inaccurate world modeling.

How do you build RL environments that resist reward hacking?

The reward looks at how a model got there as well as the result, because rewards that check only the outcome can teach the wrong lesson. In a coding run, a model that deletes a failing test can earn the same reward as one that fixes the bug. Our TRACE benchmark cataloged 54 kinds of reward hack in code environments.

Guides

Training a model for this kind of work? Talk to us

Contact →
alignment
01