GlobeBench is a benchmark of 1,632 tasks across five domains and 19 simulation skills. Each task checks whether a model can predict the environment's response to a specific action given the full agent trajectory, and we compare the prediction with an execution-derived reference. Among 20 general-purpose LLMs and specialized text world models, GPT-6 Astra achieves the highest task success at 49.57%. No model solves even half the benchmark, and 631 tasks remain unsolved by any model.
The paper is on arXiv and was accepted to the NeurIPS 2026 IAEval workshop.
Why simulators matter
As RL scales to support model capabilities in software engineering, computer use and scientific discovery, building stable, high-fidelity sandboxes remains inefficient and hard to maintain.
Recent work has shown that language models can approximate realistic world behavior and, when used as simulators, improve downstream results, including World Modeling for Agentic RL by Patronus AI's Darshan Deshpande that was accepted to NeurIPS 2026, as well as Qwen's AgentWorld. Today's models are not good enough to stand in for reinforcement learning environments. If the simulator is wrong in ways that an agent can exploit, training with it will teach the agent the mistakes of the simulator rather than the use of the software it is simulating. Previously, there was no benchmark that showed the parts of simulation that currently break, and we built GlobeBench to explore where current shortcomings are.
An example task
An agent in a Linux shell writes and overwrites several small config files. One of them, pool.conf, ends up holding a single random label, russet-willow-54, which the agent printed earlier alongside lookalike labels from other files. The final action asks for the file's line count, byte size and checksum. The answer is 17 bytes, the 16-character label plus the trailing newline the write command added. Nine of the 20 models answered correctly. Kimi K3 and GLM 5.3 both predicted 16 bytes, so their checksums were wrong too. Four Claude models refused.
Nine of twenty models pass, and two miss by one byte
The item, replayed
- Earlier
$ printf '%s\n' russet-willow-54 > /work/etc/keepalived/pool.confpool.conf is overwritten with its final content.
- Later
$ cat /work/etc/keepalived/pool.conf
russet-willow-54The environment confirms the content.
- Last step
$ rm /work/etc/keepalived/listeners.confA distractor. listeners.conf is deleted and pool.conf is untouched.
- Final action
For /work/etc/keepalived/pool.conf, print its line count, byte count, and checksum = (trailing number of first line * 131 + 17 * bytes + lines) mod 997.
- Gold
target=pool.conf kind=file lines=1 bytes=17 checksum=385- GPT-6 Astra
target=pool.conf kind=file lines=1 bytes=17 checksum=385Correct, an exact match.
- Kimi K3
target=pool.conf kind=file lines=1 bytes=16 checksum=368Wrong value. It counts 16 bytes and misses the newline printf writes.
- GPT-6 Sol
sh: arithmetic syntax errorNo state fields. It predicts a crash, but the shell runs the command and prints the stats line.
Why 17 bytes. The file holds russet-willow-54 and the newline printf writes, so lines=1, bytes=17 and the checksum is (54 * 131 + 17 * 17 + 1) mod 997 = 385.
How GlobeBench works
Each item pairs a trajectory and a final action with the gold next state, which we obtain deterministically by executing the action in the item's backend. The model sees only the trajectory and the final action, and its prediction is scored against the gold, which it never observes.
We generate 1,137 of the 1,632 items by designing the trajectory ourselves. The remaining 495 are adapted from executable open-source backends such as BFCL, TextWorld, τ²-bench and SymPy. Every item passes 12 automatic quality checks that validate answer consistency with the visible history, ensure that the gold is reproducible, and that the item is unambiguous and not a duplicate.
Each item tests one of 19 skills, which map to four capabilities, State Coherence, Causal Correctness, Interface Fidelity and Constraint Enforcement. A skill is a specific behavior of the environment, such as remembering a value that was written many turns earlier. Pass rates range from about 6% on Schema Conformance to 66% on Content Repetition.
Schema Conformance is the hardest skill, at about 6%
The four capabilities
- State Coherence
Predictions stay consistent with established entities and facts in the prior context.
- Causal Correctness
The predicted next state reflects what the action actually does.
- Interface Fidelity
Predictions match the real system's output, including the same fields, keys and error classes.
- Constraint Enforcement
Valid actions execute, and actions that violate an established constraint are refused.
Capabilities each skill tests
| Skill | Capabilities | Items |
|---|---|---|
| Schema Conformance | Interface Fidelity, Constraint Enforcement | 200 |
| Contradiction Detection | State Coherence | 110 |
| Access Control | State Coherence, Interface Fidelity, Constraint Enforcement | 224 |
| Sycophantic Confirmation | State Coherence, Constraint Enforcement | 92 |
| Format Stability | State Coherence, Interface Fidelity | 69 |
| Read and Write Persistence | State Coherence, Causal Correctness | 207 |
| Error Fidelity | Interface Fidelity, Constraint Enforcement | 41 |
| Computational Accuracy | State Coherence, Causal Correctness, Interface Fidelity | 197 |
| Partial-Failure Semantics | Causal Correctness, Interface Fidelity, Constraint Enforcement | 71 |
| Referential Integrity | State Coherence, Causal Correctness, Interface Fidelity | 50 |
| Quota Enforcement | State Coherence, Causal Correctness, Constraint Enforcement | 82 |
| Temporal Ordering and Consistency | State Coherence, Causal Correctness, Interface Fidelity | 43 |
| Transition Correctness | Causal Correctness | 25 |
| Side-Effect Propagation | State Coherence, Causal Correctness, Constraint Enforcement | 32 |
| Replay Determinism (seeded) | State Coherence, Causal Correctness | 41 |
| Under- and Over-Erroring | Causal Correctness, Constraint Enforcement | 112 |
| Rule Enforcement | State Coherence, Constraint Enforcement | 7 |
| Irreversibility Handling | State Coherence, Causal Correctness | 18 |
| Content Repetition | State Coherence, Interface Fidelity | 11 |
We benchmarked 20 models, including 18 general-purpose large language models and two built specifically as digital world models. A response can appear correct but carry the wrong numbers. We score whether the state is correct rather than whether the answer resembles one. Deterministic gates settle only those predictions that provably match or contradict the correct answer, and we delegate the rest, which require judgment, to an LLM judge. Each model is allowed one answer per item and is not provided examples in the prompt or an opportunity to reroll. The 18 general models were run at their provider's high reasoning setting, and if a model refused to attempt an item, this was counted as a failure.
What we found
GPT-6 Astra is the most performant model on GlobeBench with a 49.57% success rate, or 809 items. This is followed by Grok 4.6 at 28.13%, Claude Opus 5.5 at 26.35%, and Qwen 3.8 Max at 25.31%. Astra's lead over the second-place Grok is 350 total items.
GPT-6 Astra solves about half, and no other model reaches 30%
Four Claude models refused 3,696 answers, which were counted as failures. Nearly all refusals come from Anthropic's safeguards, which block requests that could enable cyber harm, such as malware, and can also block benign cybersecurity work. Despite this, Claude Opus 5.5 ranks third while refusing 41% of items.
Claude Opus 5.5 was successful on 430 of the 957 items it did not refuse, or roughly 45%. The items Claude refused may be disproportionately easier or harder than the rest, so the 45% isn't directly comparable to scores achieved by other models.
What makes an item hard
We record two properties of each item. Reach-back measures how far back the deciding evidence lies, and impact radius how far the action's effects reach. They correspond to the two directions in which a model that tracks state can fail, backward through the history and outward through the state.
At L0 the action changes no state. At L1 and L2 it changes only entities that it names (one at L1, two or more at L2), whereas at L3 it changes at least one entity that it does not name. Mean success is 43.49% at L0, 27.12% at L1, 27.54% at L2 and 5.10% at L3, and every model scores lowest at L3. Almost all L3 items are synthetic long-horizon audits, which are L3 by construction, so the low rate mostly reflects how hard those audits are.
Success falls to about 5% when an action changes entities it does not name
We find that when the deciding fact is within the last 40 turns, models achieve success about 42% of the time. When it is before this threshold, that number drops to 11%. All of the tested models follow this pattern. Figure 5 walks through one of these long audits step by step, where otherwise high-performing models completed correct math using a misremembered input.
Past 400 turns, the other 19 models pass at most 8.3%
One long audit, replayed
- Turn 0
Every acknowledgement exposes only applied/ignored, never current capability values.The contract. Each sponsor's vote weight and the resource's state are computed from a hidden number, its capability, and the environment never prints it.
- Turn 34
{"capability":819,"node":"resource-quorum-resource-33","op":"register_resource"}The resource is registered with capability 819.
- Later turns
The capability changes 48 more times through reconcile and restore operations. The last prefix action sets it to 89.
- Final action
{"request":"round-a-00-00","sponsors":[11,2,16],"threshold":5,"from_state":"active","to_state":"suspended"}Allow or deny the request, with each sponsor's weight.
- Gold
{"effective_capability":89,"sponsor_weights":[1,3,1],"vote_total":5,"observed_state":"active","allowed":true}- GPT-6 Astra
{"effective_capability":89,"sponsor_weights":[1,3,1],"vote_total":5,"observed_state":"active","allowed":true}Correct, an exact match.
- Grok 4.6
{"effective_capability":262,"sponsor_weights":[4,1,4],"vote_total":9,"observed_state":"suspended","allowed":false,"reason":"stale_state"}Wrong value. Its weights and state follow correctly from 262, so it denies a request that should pass.
- Others
GPT-5.5, GPT-5.6 Sol and GPT-6 Sol report a capability of 25, 992 and 877.
Why it fails. The math follows from the number each model remembers. Only Astra remembers the right number.
Accuracy is lowest where an action's effects reach dependent objects or where reach-back exceeds 40 turns. On the 952 items with both, GPT-6 Astra passes 33.5% against a median of 2.2% for the other 19 models, which accounts for 272 of its 350-item lead over Grok 4.6. Astra also wrote far less on these questions, a median of 8,300 tokens to Grok's 39,900.
631 of the 1,632 total questions were not solved by any of the 20 models tested, about two in five. Almost all of these examples are lengthy and difficult and contain actions that ripple into things that aren't explicitly named, or have their deciding turn far back in a trajectory's history. Only four items were solved by all 20 models.
How predictions fail
When models produce incorrect answers, the most common way they appear is a confident, well-formatted answer with at least one wrong value in it. This is how 53% of all wrong answers were returned. In one example, after a partial update to a store's database, GPT-5.6 Sol produced the correct product IDs and format, but reported a stock value of 5 where execution produced 6. About one quarter of incorrect answers add or drop fields, but without a wrong value. The rest don't give a state to compare at all and provide answers in prose or other incorrect formats. Open-weight models are more likely to provide an incorrect value, while closed models are more likely to leave out or add fields. A wrong answer said confidently will be undetectable by an agent in training. Simulators must achieve an exceptional level of success to be used in training.
Most failures report a wrong value, not a wrong shape
One answered failure of each type
Wrong value
- Action
sql: UPDATE products SET stock = stock - CASE WHEN stock > 10 THEN 2 ELSE 1 END WHERE id IN (100017, 100018, 904927) RETURNING id, stock- Gold
[{"id": 100017, "stock": 6}, {"id": 100018, "stock": 9}]- Predicted
[{"id": 100017, "stock": 5}, {"id": 100018, "stock": 9}]
Why it fails. Stock 7 is not over 10, so the CASE subtracts 1 and leaves 6.
Structure only
- Action
api.resolve_reference: {"reference_file": "control.ref"}- Gold
{"reference": "control.ref", "target": "notifier-backpressure.conf", "file_content": "lumen-umber-56"}- Predicted
{"error": "dangling reference: 'lumen-umber-56' not found"}
Why it fails. The three gold fields are missing and an invented error field takes their place, a schema substitution.
No state fields
- Action
game: craft banana- Gold
banana is full; crafting quota exceeded.- Predicted
Crafted one charge for banana (2/2).
Why it fails. It predicts that crafting succeeds, but banana was already stocked to its capacity of 2, so the game refuses with a quota error.
Limits and what we release
Each item tests only one transition after a fixed prefix, so we do not measure how a simulator's own errors compound over a rollout. Task success is also binary over the whole next state, so a long audit fails if any single value is wrong. Thus, a high score on its own does not mean a simulator is safe to train on.
Today we are open-sourcing the entirety of our work with GlobeBench. All 1,632 questions with their correct answers and tags, the code that runs the tests, the grader, and every model's answer and every judge's decision. Anyone can rescore this test without calling a model at all. These questions can be used by people building training environments to understand whether their simulator is safe to use, and by teams building world models who want to understand which skills their model gets wrong. Anyone can run their own model on GlobeBench using the code on GitHub.
