Last month we built a post-training stack to measure what our tasks are worth, and training with it fixed about a fifth of an open model's failures. A score like this tells you a model has improved. It doesn't tell you why the model was failing in the first place. So this time we looked inside.
When a model, or an agent built on one, gets a large, busy system wrong, a wrong answer can mean two very different things. The model may never have worked out the answer at all. Or it may have worked it out internally and then said something else. From the outside these look identical. Inside the model they are not, and they call for different fixes and different monitors.
We built a generator of tasks in which a model reasons about a software system from its text description: a synthetic e-commerce platform's service dependencies, pricing rules, shared settings, stock and change logs, the kind of state a coding or operations agent has to keep straight. Five independent complexity knobs each turn up one kind of difficulty, and every task has an exact answer. We found where Claude models break, then opened up five open-weight models to see what was happening inside when they failed. On one task, rate of change (tracking which settings changed since a snapshot), every model we probed gets answers wrong that its own activations get right. On a sorting score where 0.5 is a coin flip and 1.0 is perfect, a simple linear probe on the activations beats the model's own answers in all five models, by 0.07 to 0.25: for Qwen3-8B, 0.91 against 0.66. On another task, service topology (following failures down long dependency chains), we find no sign the answer was ever computed: probes on the activations do no better than the model's own answers or than the input words alone (for Qwen3-8B, at most 0.74 against 0.69 for the model and 0.71 for the words alone).
Rate of changeTopology, all questions
Gap in sorting score (AUROC) on held-out data, probe on activations minus the model’s own answer
Bar chart of the gap in sorting score on held-out data, a probe on the activations minus the model’s own answer, for each model, on all topology questions and on rate of change. Qwen3-4B, topology +0.05, rate of change +0.23. Qwen3-8B, topology 0.00, rate of change +0.25. Qwen3.6-27B, topology +0.03, rate of change +0.10. Qwen2.5-72B, topology +0.03, rate of change +0.07. OLMo-2 7B, topology +0.04, rate of change +0.25.
On rate of change, where the answer is already inside, the fix should not need to teach the task from scratch, only to make the model say what it already knows. Tracing one answer through Qwen3-8B shows where it gets lost: by layer 24 the right answer has reached the position where the model produces its answer word, but the remaining layers don’t turn it into that word. So we retrained small blocks of layers before that point, one at a time. Every block we tried before layer 24 works; with layers 14–17 the model's own answers go from 24% to 96–99% right on held-out logs, in each of three runs. On topology's long chains there seems to be nothing inside to link: the answer has to be computed. One pass through the model works out only a few hops (the probes partly find the answer at 2–3 hops and lose it beyond that), so we let the model think, writing the chain out one hop at a time. That fixes it, but only when the model is large enough to get every step right: at 13+ hops, with thinking on, accuracy goes from 6% (Qwen3-8B) to 86% (Qwen3-235B).
Five knobs, one at a time
Every task describes a synthesized e-commerce platform in text and asks one question with a single answer, computed by simulating the system. Names are random, so the model can't lean on what it knows about real systems. Within each knob, every prompt is padded with irrelevant filler lines to the same length as the hardest level, so when accuracy falls it is the complexity that changed, not the prompt length. Each knob turns up one kind of complexity while the others stay at their easiest setting.
| Knob | The task | Turned up |
|---|---|---|
| Topology | One service is down; which others fail? A service fails if anything it calls fails. | 4 to 128 services, longer chains, second paths and loops |
| Behavior | What is an item's final price at checkout? | A fixed price becomes stacked discounts, fees, promotions and rules that look back in time |
| Actors | The final value of one shared setting after many edits, each read, changed and written back | 1 to 16 writers at once (engineers, scripts or services), so more edits silently overwrite each other |
| Observability | How many units of an item are in stock now? | Direct counts give way to clues: orders with no quantity, damaged deliveries, a later audit |
| Rate of change | Which settings now differ from a snapshot taken earlier? | 0 to 32 changes, most of which cancel out (changed then changed back, or set to the same value) |
The knobs are separated by design but not orthogonal: each lives in its own task with its own answer format, and several of them stress the same underlying ability, keeping track of state.
Topology
Arrow means “calls”. A service fails if anything it calls fails, so the failure travels against the arrows.
What the model sees
service checkout_eu: calls payments_v2, cartservice payments_v2: calls fraud_checkservice cart: calls catalog_coreservice search: calls catalog_coreservice fraud_check: calls (nothing)service catalog_core: calls (nothing)Question
Service catalog_core is down. Which other services fail?
Right answer
cart, search, checkout_eu
Why
cart and search call it directly. checkout_eu calls cart, so it fails one hop later. payments_v2 and fraud_check are safe.
Easiest level
4 services, short chains
Turned all the way up
128 services, long chains, second paths and loops
Rate of change
What the model sees
billing.quota = 40 · cart.ttl = 25fraud.limit = 61 · search.depth = 3the agent reads every value (its snapshot)
billing.quota set to 71billing.quota set to 40cart.ttl set to 25fraud.limit set to 9search.depth set to 5search.depth set to 4search.depth set to 3Question
Which settings now differ from the agent’s snapshot?
Right answer
fraud.limit only
Why
billing.quota changed and came back, cart.ttl was set to the value it had, search.depth changed twice and came back.
Easiest level
No changes after the snapshot
Turned all the way up
32 changes for 16 settings. 3 truly differ, the rest is churn
Two worked example tasks, one above the other. Topology. A graph of six services, where an arrow means calls. checkout_eu calls payments_v2 and cart, payments_v2 calls fraud_check, cart and search call catalog_core. catalog_core is down, cart and search fail, checkout_eu fails on the second hop, payments_v2 and fraud_check are OK. The model sees no picture, only one line per service, service checkout_eu calls payments_v2, cart; service payments_v2 calls fraud_check; service cart calls catalog_core; service search calls catalog_core; service fraud_check calls (nothing); service catalog_core calls (nothing). Question, Service catalog_core is down. Which other services fail? Right answer, cart, search, checkout_eu. Why, cart and search call it directly. checkout_eu calls cart, so it fails one hop later. payments_v2 and fraud_check are safe. Easiest level, 4 services, short chains. Turned all the way up, 128 services, long chains, second paths and loops. Rate of change. What the model sees, the settings billing.quota = 40, cart.ttl = 25; fraud.limit = 61, search.depth = 3, then the agent reads every value (its snapshot), then seven edits, billing.quota set to 71, billing.quota set to 40, cart.ttl set to 25, fraud.limit set to 9, search.depth set to 5, search.depth set to 4, search.depth set to 3. Question, Which settings now differ from the agent’s snapshot? Right answer, fraud.limit only. Why, billing.quota changed and came back, cart.ttl was set to the value it had, search.depth changed twice and came back. Easiest level, No changes after the snapshot. Turned all the way up, 32 changes for 16 settings. 3 truly differ, the rest is churn.
Where Claude models break
We ran Claude Haiku 4.5 and Sonnet 5.5, with thinking off and on, on 30 tasks at each difficulty level of each knob (for example 4, 8, 16, 32, 64 and 128 services for topology), about 4,000 tasks in all. Answers are scored by exact match on a final ANSWER line. For the follow-up tests below we added Opus 5.5. Sonnet and Opus could not be fully kept from thinking, so only Haiku has a clean off versus on comparison.
- Topology is the hardest knob: it is the only one on which Sonnet slips (80% at 128 services), and Haiku stays low (33% without thinking, 57% with it). A list must match exactly, so one dropped or extra name scores zero, and at larger sizes Sonnet's misses are mostly dropped names.
- Actors break Haiku without thinking (97% with one writer, 27–43% with two or more); thinking brings it to about 80%. Sonnet stays near 100%.
- Rate of change breaks only Haiku without thinking (100% at one change, 40% at 32), mostly through missed real changes.
- Behavior and observability are flat at the original levels. Harder behavior levels break Haiku without thinking once a price rule looks back three events (37%); Sonnet stays at 100%.
Sonnet 5.5Haiku 4.5, thinking onHaiku 4.5, thinking off
Line chart of topology accuracy by number of services, 4, 8, 16, 32, 64 and 128. Sonnet 5.5, 100%, 100%, 100%, 100%, 90%, 80%. Haiku 4.5, thinking on, 100%, 97%, 100%, 100%, 73%, 57%. Haiku 4.5, thinking off, 100%, 97%, 93%, 67%, 60%, 33%.
Size or depth? In the topology curve, more services also means longer failure chains. We measure a chain in hops, so a service 5 hops from the broken one fails only through a chain of 5 failures. Asked as yes/no questions (does service S fail?), an easier format than the list answers above, growing the system from 32 to 1,024 services with the chain fixed at 5 hops barely matters (Haiku without thinking at 87% or above, every other setup at 93% or above). On questions whose answer is yes, lengthening the chain at 128 services takes Haiku without thinking from 100% at 1–3 hops to 18% at 12 hops and about 8% beyond. With thinking Haiku stays at 96–100%, and Sonnet and Opus 5.5 answer every question correctly.
Sonnet 5.5 and Opus 5.5Haiku 4.5, thinking onHaiku 4.5, thinking off
Size
Depth
Two line charts of accuracy on yes or no topology questions. Size, services 32, 128, 256, 512 and 1,024 with the chain fixed at 5 hops. Sonnet 5.5 and Opus 5.5, 100%, 100%, 100%, 100%, 100%. Haiku 4.5, thinking on, 100%, 100%, 100%, 97%, 93%. Haiku 4.5, thinking off, 97%, 100%, 87%, 90%, 90%. Depth, hops away 1 to 3, 5 to 7, 9, 12 and 13 to 17 at 128 services. Sonnet 5.5 and Opus 5.5, 100%, 100%, 100%, 100%, 100%. Haiku 4.5, thinking on, 100%, 100%, 100%, 100%, 96%. Haiku 4.5, thinking off, 100%, 73%, 67%, 18%, 8%.
What breaks even the strongest models. We then looked for a setting that breaks the strongest setups too, keeping answers to one number so a miss cannot come from dropping one name from a long list. Counting the failing services breaks every model, Opus included; following 1,024 edits by 128 writers breaks Haiku and Sonnet and pulls Opus down to 80%:
| Task (30 per cell) | Haiku, thinking | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|
| Count failing services among 512 | 3% | 13% | 53% |
| Count failing services among 1,024 | 0% | 7% | 40% |
| Final value after 1,024 edits by 128 writers | 3% | 17% | 80% |
The failure modes differ. Haiku and Sonnet undercount: their answers are too low on 25 to 29 of 30 tasks and almost never too high (Sonnet answered 55 where the truth was 443), and they write less working on the bigger problem. Opus traces nearly everything: when it is wrong on counting it is typically off by 2% at 512 services and 4% at 1,024, but exact scoring counts a near miss as wrong. Three other extremes did not break the strong setups: chains up to 17 hops, a 1,024-service system asked about one service at a time, and chains combined with changing settings (a pilot, without Opus). What breaks them is not one kind of complexity but the amount of state that must be kept exactly right at once: hundreds of services to classify and count, or a thousand interleaved edits to follow.
Looking inside: the model knows more than it says
Claude's internals are closed, so we turned to open models: Qwen3-4B, Qwen3-8B, Qwen3.6-27B, Qwen2.5-72B and, as a second family, OLMo-2 7B. Each task is turned into yes/no questions about one setting or service at a time ("Is this setting different from the snapshot?", "Does service S fail?"). For each question we record two things from the same forward pass: the model's probability of answering yes, read from its first output word, and its activations at the last prompt token at every layer. A probe is a logistic regression on one layer's activations, trained on 210 logs and tested on 90 it never saw. We pick the layer using the training logs alone (layer 26 of 36 for Qwen3-8B), and thinking is off throughout. Because the probe and the model each score the same held-out questions, the two can be compared directly.
Rate of change: knows but doesn't say. The Qwen models answer "changed" for almost every setting touched after the snapshot, including 91–100% of settings that changed and then changed back. Yet a probe reading the same model's activations sorts truly changed from unchanged settings far better than the model does:
| Model (90 held-out logs, 1,440 questions) | Model's own score | Probe on its activations | Probe right when the model is wrong |
|---|---|---|---|
| Qwen3-4B | 0.74 | 0.97 | 96% of 1,028 (floor ~81%) |
| Qwen3-8B | 0.66 | 0.91 | 91% of 1,091 (floor ~81%) |
| Qwen3.6-27B | 0.88 | 0.98 | 94% of 484 (floor ~80%) |
| Qwen2.5-72B | 0.92 | 0.99 | 97% of 868 (floor ~81%) |
| OLMo-2 7B | 0.61 | 0.86 | 60% of 270 (floor ~19%) |
The first two columns are sorting scores (AUROC). Each asks how often a truly changed setting is ranked above an unchanged one, using the model's probability of answering yes in the first column and the probe's score in the second. 0.5 is a coin flip and 1.0 is perfect. The floor in the last column is what a probe that knows nothing would score, given each model's mix of mistakes: almost all of Qwen's mistakes are on unchanged settings, so a probe that says "changed" as often as settings really change would already be right about 81% of the time. OLMo fails the other way around, answering "not changed" to everything, so its floor is 19%. The sorting scores need no such adjustment, and they are the stronger evidence.
Like a smoke alarm that also rings for toast, not just for fires. The models' low sorting scores come from saying "changed" too often. Qwen3-4B says "changed" for 100% of settings that truly changed, but also for 98% of settings changed and then changed back, 100% of those changed twice and back, 81% of those set again to the same value and 79% of those never touched after the snapshot. Only 29% of its yes/no answers are right: it never misses a real change, but when it says "changed" you still don't know. Its confidence still leans slightly the right way, which is why its sorting score of 0.74 is better than its 29% of right answers. The probe, reading the same model, mostly rings only for the real fires with a sorting score of 0.97.
Truly changedNot changed, a false alarm
Bar chart of how often Qwen3-4B answers "changed", by what happened to the setting. Truly changed, 100%. Changed, then changed back, 98%. Changed twice, then back, 100%. Set again to the same value, 81%. Never touched after the snapshot, 79%. 29% of its yes or no answers are right.
Is the probe fooling us?
A probe is itself a trained classifier, so the obvious worry is that the probe, not the model, does the work. The activations hold the setting's value at the snapshot (say 45) and its value now (say 45). The probe could check whether those two numbers are equal by itself, even if the model never did. We ran the checks a skeptic would ask for, all on held-out data:
| Check | Rules out | Result |
|---|---|---|
| Probe trained on shuffled labels | a leak in the setup | 0.50, a coin flip |
| A random label for each setting name | a probe that memorizes names | 74%, worse than always guessing the majority (82%) |
| The same probe on a randomly initialized Qwen3-8B | solving the task from the text alone | 0.50 at every layer |
| Probe at the first layer (only the words) | the answer sitting in the text itself | coin flip |
| 95% intervals over held-out logs, truly changed vs changed back | a lucky split | probe 0.83–0.88 vs model 0.58–0.65 (Qwen3-8B) |
| A second model family, OLMo-2 7B | a Qwen quirk | probe 0.80 vs model 0.51 on truly changed vs changed back |
| Probe trained without any canceled change | the probe needing to see those cases | partial: changed back 0.74, changed twice and back 0.50 (Qwen3-8B); 0.88 and 0.70 (Qwen3-4B) |
The first five pass: the probe reads what the trained model computed. The last is the honest limit. The signal is not one universal "is this different from the snapshot?" detector; a probe never shown a setting changed twice and back cannot spot those in Qwen3-8B.
On long chains, no sign the answer is computed
Topology: can't. The hardest topology questions are about services that do fail, but only through a chain of 4 or more failures from the broken one. For those, we found nothing extra inside. On Qwen3-8B the model's own score is 0.69. To tell whether the model worked anything out, we also compare against the words alone: a probe on the very first layer, which sees the input words before any layer has combined them. It can still beat a coin flip from surface cues in the text, such as how many services the asked one calls, and gets 0.71. That is the score with no reasoning at all. Probes deeper in the model, linear or small neural nets at three positions (the last token, the service's name in the question, and its own line in the manifest), reach only 0.71–0.74: the layers add almost nothing (only 48 far failures in the test set, so each score is about ±0.08). On rate of change, by contrast, the first-layer probe is at chance (0.50) and the deeper probe reaches 0.91: there the layers did the work. The larger Qwen models say "does not fail" for 96% of far failures and their probes correctly flag only 27–50%; OLMo's probe is at chance there. We find no sign that the answer was ever computed.
Bar chart of Qwen3-8B sorting scores from 0.5, a coin flip, to 1.0. Long chains, far failures 4 or more hops away. The model's own score, 0.69. Words alone, a probe at the first layer, 0.71. Probe at the last token, 0.71. Probe at the service's name in the question, 0.74. Probe at its own line in the manifest, 0.71. Rate of change, the same model. The model's own score, 0.66. Probe on its activations, 0.91.
Two kinds of failure. From the outside both look like a wrong answer. On rate of change the right answer is computed inside and fails to come out: the model knows but doesn't say, so the fix should connect what it knows to its answer. On long chains it was never computed: the model can't, so the fix should give it room to compute. The next section follows the first kind through the model.
Following one answer through the model
If the model has the answer, where does it get lost? We built 48 pairs of logs that are identical except for one setting's last value. In Log A the setting ends at a new value, so the answer is yes. In Log B it returns to its snapshot value, so the answer is no, and Qwen3-8B gets it wrong:
| Log A, changed | Log B, changed back | |
|---|---|---|
| Example | 45 at the snapshot, then 71, then 63 | 45 at the snapshot, then 71, then 45 |
| Probe at the answer position, layer 26 | +1.7 (≈85% "changed") | −4.1 (≈2% "changed") |
| Model's yes-minus-no score | 8.6 (≈99.98% "yes") | 7.6 (≈99.95% "yes") |
Inside, both are right. In what it says, Log B is wrong. To trace the information we ran Log B but replaced the model's internal state at one group of positions and one layer with Log A's (shown in the following picture), then measured how far the probe's reading moved toward Log A (shown in the Table):
Two grids with the same rows and columns, Log A, which ends at 63, above Log B, which ends at 45. Rows are layers 26, 24, 22, 20, 18, 14 and 2. Columns are the number, the question and the answer position. An arrow copies Log A’s state at the answer position, layer 24, into the same cell of Log B. A probe reads Log B’s answer position at layer 26. Each Log B cell shows the share of the probe’s reading that moved toward Log A when only that cell was swapped. Layer 2, number 95%, question 0%, answer position 0%. Layer 14, number 75%, question 2%, answer position 0%. Layer 18, number 65%, question 18%, answer position 2%. Layer 20, number 41%, question 45%, answer position 1%. Layer 22, number 9%, question 69%, answer position 18%. Layer 24, number 2%, question 4%, answer position 91%.
| Layer | Swap the number | Swap the question | Swap the answer position |
|---|---|---|---|
| 2 | 95% | 0% | 0% |
| 14 | 75% | 2% | 0% |
| 18 | 65% | 18% | 2% |
| 20 | 41% | 45% | 1% |
| 22 | 9% | 69% | 18% |
| 24 | 2% | 4% | 91% |
The difference starts on the digits of the last value (95% at layer 2), moves to the question's words around layers 18–22 (69% at layer 22), and from layer 24 sits at the answer position: swapping in Log A's state there alone recovers 91% of the probe's reading, while swapping the number or the question at that layer recovers almost nothing (2% and 4%). By layer 24, nearly everything the model has worked out about whether the setting changed is gathered in the one spot where it produces its answer word. The pattern is stable across the 48 pairs (95% intervals: 87–98%, 63–77% and 88–95%) and repeats in a second, smaller Qwen model, Qwen3-4B (98%, 70% and 100%).
So the knowledge reaches exactly where the answer is produced, and still doesn’t become the word. One reason is geometric: the direction the probe reads and the direction that decides yes or no are almost at right angles (a cosine of 0.02, about 89°), so the yes/no read-out barely uses the direction where the knowledge sits.
A warning light, before any fix
If the answer is inside, it can flag the model's own mistakes. For each yes/no answer the model gave, we computed a suspicion score, how likely that answer is to be wrong, in three ways:
- The model's own confidence: 100% minus how sure the model is of its answer. It needs no look inside, but it misses answers the model gives confidently and wrongly.
- An error probe: a second probe on the same activations, trained to predict whether the model's answer is wrong.
- The truth probe against the answer: how far the truth probe's reading (changed or not) is from what the model said. A big gap means one of them is wrong, and it is usually the model.
Each light is scored by how well it ranks wrong answers above right ones, not by how confident anything is. A score of 1.0 means every wrong answer gets a higher suspicion than every right answer; 0.5 means that, for a random wrong-right pair, the wrong one is higher only half the time, which is a coin flip.
| Task and model | Warning light from the model's confidence | From the error probe | From truth probe vs answer |
|---|---|---|---|
| Rate of change, Qwen3-4B | 0.63 | 0.96 | 0.96 |
| Rate of change, Qwen3-8B | 0.52 | 0.91 | 0.91 |
| Topology, Qwen3-4B | 0.72 | 0.69 | 0.79 |
| Topology, Qwen3-8B | 0.74 | 0.67 | 0.75 |
Where the model knows (rate of change), the probes flag its mistakes well: of the 20% of answers they flag as most suspicious, 98–100% are truly wrong, against 71–76% for a random 20%, since that share of all answers is wrong anyway. Picking the 20% the model itself is least sure of does no better than random: its doubts don't point to its mistakes, because it is about as sure of its wrong answers as of its right ones (Qwen3-4B: a median of 97% sure when wrong, 99% when right). Where it can't (topology), the probes are no better than its confidence. Each light works only on the task it was trained on (0.52–0.61 on the other).
Fixing each failure
Steering shifts every answer the same way. Steering adds the same arrow to every question's activations at the answer position, at layer 28, a few layers after the knowledge reaches it (results were the same at layers 24 and 32):
h′ = h + α·σ·v̂
where v̂ is the direction of the push, scaled to length 1; σ is how much questions naturally vary along that direction; and α is the push size in those units.
We tried three directions, each pushed two ways: far beyond anything natural (11 to 13 times the gap between changed and unchanged settings), and all by the same raw amount at layers 24, 28 and 32:
- The probe's own direction: nothing happens, even at α = 16. At the same raw size, it moves 3% of answers.
- The average difference from unchanged to changed settings: every answer flips at α = 8. At the same raw size, it moves 93% of answers.
- A random direction, as a control: almost nothing happens. At the same raw size, it moves 2–4% of answers.
Here is why (illustrated below). Along the probe's direction, questions vary little (σ = 0.70), yet changed and unchanged settings separate cleanly. Along the direction the answer depends on, questions vary a lot (σ = 35.4), and the two groups overlap.
Truly changed settingUnchanged, before the pushUnchanged, after the push
Push 1, along the probe’s direction
Push 2, along the average difference
Two cartoon scatter plots of questions in two of Qwen3-8B’s internal numbers at layer 28. The vertical axis is the probe’s direction, along which questions vary little (sigma 0.70). The horizontal axis is the direction the answer depends on, along which they vary a lot (sigma 35.4). The two are nearly at right angles, cosine 0.02. Unchanged settings sit below the probe’s line and truly changed ones above it, and the two groups overlap left to right. Right of the answer’s line the model answers changed. Push 1, a push along the probe’s direction at alpha 16 moves the unchanged settings straight up across the probe’s line, and the model still answers unchanged. Push 2, a push along the average difference at alpha 8 moves them up and to the right across the answer’s line, and the answer flips to changed.
The probe's direction and the direction the answer reads (the output layer's "yes" minus "no") are nearly at right angles, with a cosine of 0.02. So a push along the probe's direction changes what the probe reads but leaves the answer untouched: in the figure, it crosses the probe's line but never the answer's. Pushing along the average difference does move the answer, but it moves every question the same way. Pushed 4σ toward "not changed" (α = −4), the model's average probability of "changed" drops to 15% for truly changed settings and 12% for changed-back ones, so it still can't tell them apart. Steering changes how often the model says "changed", not whether it is right.
For "knows but doesn't say": retrain a few middle layers. The probe says the answer is already computed, and the swap experiment says it reaches the answer position by layer 24 but does not become the output word. So the repair should connect the two before that hand-off, not teach the task again. We retrained one block of four layers of Qwen3-8B at a time, everything else frozen: small enough to show where in the model a fix works, and the same size at every depth so the blocks compare fairly. Each block was trained on 6,400 questions from fresh logs, three runs per block:
| Block retrained | Held-out accuracy, three runs |
|---|---|
| None (original) | 24% |
| Layers 4–7 | 94%, 92%, 94% |
| Layers 14–17 | 99%, 96%, 99% |
| Layers 20–23 | 94%, 94%, 97% |
| Layers 24–27 | 57%, 51%, 77% (75% with 3× training) |
| Layers 32–35 | 45%, 47%, 60% (54% with 3×, 52% with a 10× learning rate) |
Every block before layer 24 fixes it reliably; later blocks learn it slowly or barely, consistent with the hand-off we saw in the swap experiment. The layer 14–17 fix generalizes: 98% on harder logs, 95% on a reworded question, 93% and 97% on logs written as JSON events or plain sentences that it never saw in training (the base model scores 23–25% there), and its loss on 300 general chat replies does not get worse (2.013 to 1.987). The fix is spread out but small: no single layer or attention head group carries it, the MLPs carry most of it, each weight matrix moves by only 0.2–0.3% of its size, and keeping the top 256 directions of each weight change keeps 94%.
For "can't": thinking, if the model is big enough. Here nothing is inside to connect, so the fix has to add computation. A single forward pass, one trip through the model's layers to produce one word, can work out only a few hops. In the larger Qwen models, the answer is partly there inside at 2–3 hops (a probe gets 63–78% of failing services right, the model only 22–48%) and fades beyond that. Thinking lets the model write the chain out one hop at a time, each written step feeding the next pass. With thinking on, Qwen3-8B rises from 68% to 91% on 1,937 topology questions (answers that run out of room count as wrong).
But every step has to be right. On harder graphs with chains up to 22 hops, at 13+ hops Qwen3-8B with thinking reaches only 6%, even while writing thousands of reasoning tokens; Qwen3-235B-Thinking reaches 86% (99% at one hop). The large model fits a small error per hop adding up along the chain (0.99¹³ ≈ 88%). The small model falls much further than that, suggesting its steps get less reliable as the chain, and its own notes, grow longer. Thinking removes the one-pass limit, but only for a model whose steps are reliable.
Accuracy with thinking on, graphs with chains up to 22 hops
Bar chart of accuracy with thinking on, on graphs with chains up to 22 hops. Qwen3-8B, 13 or more hops, 6%. Qwen3-235B-Thinking, 13 or more hops, 86%. Qwen3-235B-Thinking, one hop, 99%.
What about Claude?
Does Claude fail the same way? We can't look inside it, but we can ask it the same 640 yes/no settings questions. Claude gets them right, Haiku 96% and Sonnet 99.5%, because both reason before answering: Haiku in visible text, Sonnet in hidden thinking on most calls, even with thinking set to off. Forced to answer in one word, Haiku drops to 80% and fails the other way from Qwen, missing 71% of real changes; Sonnet stays at 99.5%. So the failure we dissected belongs to instant one-word answers from open models, and whether Haiku's one-word mistakes are "knows but doesn't say" or "can't" is something only its internals could tell.
What we learned
- Where models break. Long chains, concurrency, churn and look-back rules break Haiku until it thinks. In two stress tests that mean tracking hundreds of facts exactly, counting failing services breaks every Claude model, Opus included, and following 1,024 edits by 128 writers breaks Haiku and Sonnet and pulls Opus down to 80%.
- Looking inside. On settings, a probe reads the right answer from the activations: the model knows but doesn't say, and the probe can flag its mistakes. On long chains, no probe finds it: the model can't.
- Saying it right. Steering shifts every answer the same way. Retraining a few middle layers fixes "knows but doesn't say"; thinking fixes "can't" when the model is big enough.
- Diagnose before fixing. The same wrong answer can mean "never computed" or "computed but not used". One probe and one control tell them apart, and the diagnosis predicted which fix worked.
Limitations
The tasks are synthetic text, with one generator per task and 30 tasks per point in the Claude curves, so each point carries a 95% margin of error of about ±15 points (up to ±18 near 50%). Claude's thinking could not be fully switched off through our interface: Sonnet thought on about half of its "off" calls. The internal analysis uses Qwen and OLMo, not Claude. Patching, steering and retraining are on one model, Qwen3-8B, with three runs per retraining block. We did not vary the block size, so four layers is a choice, not a minimum.
What's next
- Does scale close the gap? Bigger models narrow it (72B: model 0.92, probe 0.99). We want to know whether scale fixes the read-out, lets one pass resolve more hops, or both.
- Do the probes work on real systems? Run them and the warning light on real configuration diffs and incident logs.
- How does the model compute "changed", and why does the answer ignore it? Find the parts of the model that compare the current value with the snapshot, and what the yes/no answer reads instead.
- Is the long-chain failure in reading the graph or in following it? Check whether the model even holds each "A calls B" link, before blaming the chaining.
Appendix: setup details
| Item | Setting |
|---|---|
| Claude models | Haiku 4.5, Sonnet 5.5, Opus 5.5 (stress tests), via a command-line interface, thinking off and on |
| Open models | Qwen3-4B-Instruct, Qwen3-8B, Qwen3.6-27B, Qwen2.5-72B-Instruct, OLMo-2 7B Instruct; thinking off |
| Probe | Standardized activations, L2 logistic regression (C = 0.05); layer chosen by 5-fold cross-validation on 210 training logs; scored on 90 held-out logs |
| Rate-of-change data | 300 logs × 16 settings = 4,800 questions per model |
| Topology data | 300 dependency lists of 8 to 128 services, about 1,900 questions per model |
| Patching | 48 log pairs, Qwen3-8B (and Qwen3-4B); probe read at layer 26; recovery = (after swap − Log B) ÷ (Log A − Log B) |
| Steering | Qwen3-8B, answer position, layer 28 (24 and 32 for matched size); 640 questions from 40 fresh logs |
| Retraining | AdamW, learning rate 1e-5, batch 8, 800 steps, class-weighted loss on the yes-minus-no score; three runs per block |



