Investigating What Open Models Knew When They Answered Questions Incorrectly

We used probes on open models to separate two kinds of failure. On one task, a probe reading the model's activations beat the model's own answers, and retraining four layers took Qwen3-8B from 24% to 96-99% accuracy. On another, we found no sign the correct answer was ever computed.

Cloud inversion, Grand Canyon from Desert View, Arizona, Nov 29, 2013. Photo: Erin Huggins, National Park Service (public domain)

Last month we built a post-training stack to measure what our tasks are worth, and training with it fixed about a fifth of an open model's failures. A score like this tells you a model has improved. It doesn't tell you why the model was failing in the first place. So this time we looked inside.

When a model, or an agent built on one, gets a large, busy system wrong, a wrong answer can mean two very different things. The model may never have worked out the answer at all. Or it may have worked it out internally and then said something else. From the outside these look identical. Inside the model they are not, and they call for different fixes and different monitors.

We built a generator of tasks in which a model reasons about a software system from its text description: a synthetic e-commerce platform's service dependencies, pricing rules, shared settings, stock and change logs, the kind of state a coding or operations agent has to keep straight. Five independent complexity knobs each turn up one kind of difficulty, and every task has an exact answer. We found where Claude models break, then opened up five open-weight models to see what was happening inside when they failed. On one task, rate of change (tracking which settings changed since a snapshot), every model we probed gets answers wrong that its own activations get right. On a sorting score where 0.5 is a coin flip and 1.0 is perfect, a simple linear probe on the activations beats the model's own answers in all five models, by 0.07 to 0.25: for Qwen3-8B, 0.91 against 0.66. On another task, service topology (following failures down long dependency chains), we find no sign the answer was ever computed: probes on the activations do no better than the model's own answers or than the input words alone (for Qwen3-8B, at most 0.74 against 0.69 for the model and 0.71 for the words alone).

Rate of changeTopology, all questions

Bar chart of the gap in sorting score on held-out data, a probe on the activations minus the model’s own answer, for each model, on all topology questions and on rate of change. Qwen3-4B, topology +0.05, rate of change +0.23. Qwen3-8B, topology 0.00, rate of change +0.25. Qwen3.6-27B, topology +0.03, rate of change +0.10. Qwen2.5-72B, topology +0.03, rate of change +0.07. OLMo-2 7B, topology +0.04, rate of change +0.25.

Figure 1. On rate of change the probe is far ahead in every model; on topology, measured over all its questions, it is barely ahead. On long chains alone (Qwen3-8B), the best probe reaches 0.74, against 0.69 for the model and 0.71 from the words alone.

On rate of change, where the answer is already inside, the fix should not need to teach the task from scratch, only to make the model say what it already knows. Tracing one answer through Qwen3-8B shows where it gets lost: by layer 24 the right answer has reached the position where the model produces its answer word, but the remaining layers don’t turn it into that word. So we retrained small blocks of layers before that point, one at a time. Every block we tried before layer 24 works; with layers 14–17 the model's own answers go from 24% to 96–99% right on held-out logs, in each of three runs. On topology's long chains there seems to be nothing inside to link: the answer has to be computed. One pass through the model works out only a few hops (the probes partly find the answer at 2–3 hops and lose it beyond that), so we let the model think, writing the chain out one hop at a time. That fixes it, but only when the model is large enough to get every step right: at 13+ hops, with thinking on, accuracy goes from 6% (Qwen3-8B) to 86% (Qwen3-235B).

Five knobs, one at a time

Every task describes a synthesized e-commerce platform in text and asks one question with a single answer, computed by simulating the system. Names are random, so the model can't lean on what it knows about real systems. Within each knob, every prompt is padded with irrelevant filler lines to the same length as the hardest level, so when accuracy falls it is the complexity that changed, not the prompt length. Each knob turns up one kind of complexity while the others stay at their easiest setting.

Table 1. The five complexity knobs, the question each task asks and what turning the knob up changes.
KnobThe taskTurned up
TopologyOne service is down; which others fail? A service fails if anything it calls fails.4 to 128 services, longer chains, second paths and loops
BehaviorWhat is an item's final price at checkout?A fixed price becomes stacked discounts, fees, promotions and rules that look back in time
ActorsThe final value of one shared setting after many edits, each read, changed and written back1 to 16 writers at once (engineers, scripts or services), so more edits silently overwrite each other
ObservabilityHow many units of an item are in stock now?Direct counts give way to clues: orders with no quantity, damaged deliveries, a later audit
Rate of changeWhich settings now differ from a snapshot taken earlier?0 to 32 changes, most of which cancel out (changed then changed back, or set to the same value)
Table 1. The five complexity knobs, the question each task asks and what turning the knob up changes.

The knobs are separated by design but not orthogonal: each lives in its own task with its own answer format, and several of them stress the same underlying ability, keeping track of state.

Two worked example tasks, one above the other. Topology. A graph of six services, where an arrow means calls. checkout_eu calls payments_v2 and cart, payments_v2 calls fraud_check, cart and search call catalog_core. catalog_core is down, cart and search fail, checkout_eu fails on the second hop, payments_v2 and fraud_check are OK. The model sees no picture, only one line per service, service checkout_eu calls payments_v2, cart; service payments_v2 calls fraud_check; service cart calls catalog_core; service search calls catalog_core; service fraud_check calls (nothing); service catalog_core calls (nothing). Question, Service catalog_core is down. Which other services fail? Right answer, cart, search, checkout_eu. Why, cart and search call it directly. checkout_eu calls cart, so it fails one hop later. payments_v2 and fraud_check are safe. Easiest level, 4 services, short chains. Turned all the way up, 128 services, long chains, second paths and loops. Rate of change. What the model sees, the settings billing.quota = 40, cart.ttl = 25; fraud.limit = 61, search.depth = 3, then the agent reads every value (its snapshot), then seven edits, billing.quota set to 71, billing.quota set to 40, cart.ttl set to 25, fraud.limit set to 9, search.depth set to 5, search.depth set to 4, search.depth set to 3. Question, Which settings now differ from the agent’s snapshot? Right answer, fraud.limit only. Why, billing.quota changed and came back, cart.ttl was set to the value it had, search.depth changed twice and came back. Easiest level, No changes after the snapshot. Turned all the way up, 32 changes for 16 settings. 3 truly differ, the rest is churn.

Figure 2. Two knobs as tasks. Top, topology, where catalog_core is down, so cart and search fail, and checkout_eu fails one hop later. Bottom, rate of change, where three settings are touched after the snapshot but end where they started, so only fraud.limit differs.

Where Claude models break

We ran Claude Haiku 4.5 and Sonnet 5.5, with thinking off and on, on 30 tasks at each difficulty level of each knob (for example 4, 8, 16, 32, 64 and 128 services for topology), about 4,000 tasks in all. Answers are scored by exact match on a final ANSWER line. For the follow-up tests below we added Opus 5.5. Sonnet and Opus could not be fully kept from thinking, so only Haiku has a clean off versus on comparison.

  • Topology is the hardest knob: it is the only one on which Sonnet slips (80% at 128 services), and Haiku stays low (33% without thinking, 57% with it). A list must match exactly, so one dropped or extra name scores zero, and at larger sizes Sonnet's misses are mostly dropped names.
  • Actors break Haiku without thinking (97% with one writer, 27–43% with two or more); thinking brings it to about 80%. Sonnet stays near 100%.
  • Rate of change breaks only Haiku without thinking (100% at one change, 40% at 32), mostly through missed real changes.
  • Behavior and observability are flat at the original levels. Harder behavior levels break Haiku without thinking once a price rule looks back three events (37%); Sonnet stays at 100%.

Line chart of topology accuracy by number of services, 4, 8, 16, 32, 64 and 128. Sonnet 5.5, 100%, 100%, 100%, 100%, 90%, 80%. Haiku 4.5, thinking on, 100%, 97%, 100%, 100%, 73%, 57%. Haiku 4.5, thinking off, 100%, 97%, 93%, 67%, 60%, 33%.

Figure 3. Topology accuracy as the system grows from 4 to 128 services, 30 tasks per point (about ±15 points). Thinking helps Haiku but does not close the gap.

Size or depth? In the topology curve, more services also means longer failure chains. We measure a chain in hops, so a service 5 hops from the broken one fails only through a chain of 5 failures. Asked as yes/no questions (does service S fail?), an easier format than the list answers above, growing the system from 32 to 1,024 services with the chain fixed at 5 hops barely matters (Haiku without thinking at 87% or above, every other setup at 93% or above). On questions whose answer is yes, lengthening the chain at 128 services takes Haiku without thinking from 100% at 1–3 hops to 18% at 12 hops and about 8% beyond. With thinking Haiku stays at 96–100%, and Sonnet and Opus 5.5 answer every question correctly.

Two line charts of accuracy on yes or no topology questions. Size, services 32, 128, 256, 512 and 1,024 with the chain fixed at 5 hops. Sonnet 5.5 and Opus 5.5, 100%, 100%, 100%, 100%, 100%. Haiku 4.5, thinking on, 100%, 100%, 100%, 97%, 93%. Haiku 4.5, thinking off, 97%, 100%, 87%, 90%, 90%. Depth, hops away 1 to 3, 5 to 7, 9, 12 and 13 to 17 at 128 services. Sonnet 5.5 and Opus 5.5, 100%, 100%, 100%, 100%, 100%. Haiku 4.5, thinking on, 100%, 100%, 100%, 100%, 96%. Haiku 4.5, thinking off, 100%, 73%, 67%, 18%, 8%.

Figure 4. Accuracy on yes or no topology questions as the system grows (Size) and as the chain gets longer (Depth). Growing the system barely matters, while a longer chain breaks Haiku without thinking. The depth panel counts yes questions only.

What breaks even the strongest models. We then looked for a setting that breaks the strongest setups too, keeping answers to one number so a miss cannot come from dropping one name from a long list. Counting the failing services breaks every model, Opus included; following 1,024 edits by 128 writers breaks Haiku and Sonnet and pulls Opus down to 80%:

Table 2. Share of 30 tasks answered exactly right in the two stress tests, counting failing services and following 1,024 edits by 128 writers.
Task (30 per cell)Haiku, thinkingSonnet 5.5Opus 5.5
Count failing services among 5123%13%53%
Count failing services among 1,0240%7%40%
Final value after 1,024 edits by 128 writers3%17%80%
Table 2. Share of 30 tasks answered exactly right in the two stress tests, counting failing services and following 1,024 edits by 128 writers.

The failure modes differ. Haiku and Sonnet undercount: their answers are too low on 25 to 29 of 30 tasks and almost never too high (Sonnet answered 55 where the truth was 443), and they write less working on the bigger problem. Opus traces nearly everything: when it is wrong on counting it is typically off by 2% at 512 services and 4% at 1,024, but exact scoring counts a near miss as wrong. Three other extremes did not break the strong setups: chains up to 17 hops, a 1,024-service system asked about one service at a time, and chains combined with changing settings (a pilot, without Opus). What breaks them is not one kind of complexity but the amount of state that must be kept exactly right at once: hundreds of services to classify and count, or a thousand interleaved edits to follow.

Looking inside: the model knows more than it says

Claude's internals are closed, so we turned to open models: Qwen3-4B, Qwen3-8B, Qwen3.6-27B, Qwen2.5-72B and, as a second family, OLMo-2 7B. Each task is turned into yes/no questions about one setting or service at a time ("Is this setting different from the snapshot?", "Does service S fail?"). For each question we record two things from the same forward pass: the model's probability of answering yes, read from its first output word, and its activations at the last prompt token at every layer. A probe is a logistic regression on one layer's activations, trained on 210 logs and tested on 90 it never saw. We pick the layer using the training logs alone (layer 26 of 36 for Qwen3-8B), and thinking is off throughout. Because the probe and the model each score the same held-out questions, the two can be compared directly.

Rate of change: knows but doesn't say. The Qwen models answer "changed" for almost every setting touched after the snapshot, including 91–100% of settings that changed and then changed back. Yet a probe reading the same model's activations sorts truly changed from unchanged settings far better than the model does:

Table 3. Rate of change on 90 held-out logs. Sorting scores run from 0.5, a coin flip, to 1.0, perfect. The drawn column shows the gap from the model's own score (gray) to the probe (teal).
Model (90 held-out logs, 1,440 questions)Model's own scoreProbe on its activationsProbe right when the model is wrong
Qwen3-4B0.740.9796% of 1,028 (floor ~81%)
Qwen3-8B0.660.9191% of 1,091 (floor ~81%)
Qwen3.6-27B0.880.9894% of 484 (floor ~80%)
Qwen2.5-72B0.920.9997% of 868 (floor ~81%)
OLMo-2 7B0.610.8660% of 270 (floor ~19%)
Table 3. Rate of change on 90 held-out logs. Sorting scores run from 0.5, a coin flip, to 1.0, perfect. The drawn column shows the gap from the model's own score (gray) to the probe (teal).

The first two columns are sorting scores (AUROC). Each asks how often a truly changed setting is ranked above an unchanged one, using the model's probability of answering yes in the first column and the probe's score in the second. 0.5 is a coin flip and 1.0 is perfect. The floor in the last column is what a probe that knows nothing would score, given each model's mix of mistakes: almost all of Qwen's mistakes are on unchanged settings, so a probe that says "changed" as often as settings really change would already be right about 81% of the time. OLMo fails the other way around, answering "not changed" to everything, so its floor is 19%. The sorting scores need no such adjustment, and they are the stronger evidence.

Like a smoke alarm that also rings for toast, not just for fires. The models' low sorting scores come from saying "changed" too often. Qwen3-4B says "changed" for 100% of settings that truly changed, but also for 98% of settings changed and then changed back, 100% of those changed twice and back, 81% of those set again to the same value and 79% of those never touched after the snapshot. Only 29% of its yes/no answers are right: it never misses a real change, but when it says "changed" you still don't know. Its confidence still leans slightly the right way, which is why its sorting score of 0.74 is better than its 29% of right answers. The probe, reading the same model, mostly rings only for the real fires with a sorting score of 0.97.

Bar chart of how often Qwen3-4B answers "changed", by what happened to the setting. Truly changed, 100%. Changed, then changed back, 98%. Changed twice, then back, 100%. Set again to the same value, 81%. Never touched after the snapshot, 79%. 29% of its yes or no answers are right.

Figure 5. How often Qwen3-4B says "changed", by what really happened to the setting. Only the first group truly changed. Across all groups, 29% of Qwen3-4B's yes or no answers are right.

Is the probe fooling us?

A probe is itself a trained classifier, so the obvious worry is that the probe, not the model, does the work. The activations hold the setting's value at the snapshot (say 45) and its value now (say 45). The probe could check whether those two numbers are equal by itself, even if the model never did. We ran the checks a skeptic would ask for, all on held-out data:

Table 4. Controls on held-out data. The last check passes only in part.
CheckRules outResult
Probe trained on shuffled labelsa leak in the setup0.50, a coin flip
A random label for each setting namea probe that memorizes names74%, worse than always guessing the majority (82%)
The same probe on a randomly initialized Qwen3-8Bsolving the task from the text alone0.50 at every layer
Probe at the first layer (only the words)the answer sitting in the text itselfcoin flip
95% intervals over held-out logs, truly changed vs changed backa lucky splitprobe 0.83–0.88 vs model 0.58–0.65 (Qwen3-8B)
A second model family, OLMo-2 7Ba Qwen quirkprobe 0.80 vs model 0.51 on truly changed vs changed back
Probe trained without any canceled changethe probe needing to see those casespartial: changed back 0.74, changed twice and back 0.50 (Qwen3-8B); 0.88 and 0.70 (Qwen3-4B)
Table 4. Controls on held-out data. The last check passes only in part.

The first five pass: the probe reads what the trained model computed. The last is the honest limit. The signal is not one universal "is this different from the snapshot?" detector; a probe never shown a setting changed twice and back cannot spot those in Qwen3-8B.

On long chains, no sign the answer is computed

Topology: can't. The hardest topology questions are about services that do fail, but only through a chain of 4 or more failures from the broken one. For those, we found nothing extra inside. On Qwen3-8B the model's own score is 0.69. To tell whether the model worked anything out, we also compare against the words alone: a probe on the very first layer, which sees the input words before any layer has combined them. It can still beat a coin flip from surface cues in the text, such as how many services the asked one calls, and gets 0.71. That is the score with no reasoning at all. Probes deeper in the model, linear or small neural nets at three positions (the last token, the service's name in the question, and its own line in the manifest), reach only 0.71–0.74: the layers add almost nothing (only 48 far failures in the test set, so each score is about ±0.08). On rate of change, by contrast, the first-layer probe is at chance (0.50) and the deeper probe reaches 0.91: there the layers did the work. The larger Qwen models say "does not fail" for 96% of far failures and their probes correctly flag only 27–50%; OLMo's probe is at chance there. We find no sign that the answer was ever computed.

Bar chart of Qwen3-8B sorting scores from 0.5, a coin flip, to 1.0. Long chains, far failures 4 or more hops away. The model's own score, 0.69. Words alone, a probe at the first layer, 0.71. Probe at the last token, 0.71. Probe at the service's name in the question, 0.74. Probe at its own line in the manifest, 0.71. Rate of change, the same model. The model's own score, 0.66. Probe on its activations, 0.91.

Figure 6. Qwen3-8B sorting scores. On far failures, probes deeper in the model score 0.71 to 0.74, within the margin of about ±0.08 of the 0.71 from the words alone, so the layers add almost nothing. On rate of change, the probe scores 0.91 against the model's 0.66.

Two kinds of failure. From the outside both look like a wrong answer. On rate of change the right answer is computed inside and fails to come out: the model knows but doesn't say, so the fix should connect what it knows to its answer. On long chains it was never computed: the model can't, so the fix should give it room to compute. The next section follows the first kind through the model.

Following one answer through the model

If the model has the answer, where does it get lost? We built 48 pairs of logs that are identical except for one setting's last value. In Log A the setting ends at a new value, so the answer is yes. In Log B it returns to its snapshot value, so the answer is no, and Qwen3-8B gets it wrong:

Table 5. Two logs that differ only in one setting's last value, with the probe's and the model's scores. The model says yes to both.
Log A, changedLog B, changed back
Example45 at the snapshot, then 71, then 6345 at the snapshot, then 71, then 45
Probe at the answer position, layer 26+1.7 (≈85% "changed")−4.1 (≈2% "changed")
Model's yes-minus-no score8.6 (≈99.98% "yes")7.6 (≈99.95% "yes")
Table 5. Two logs that differ only in one setting's last value, with the probe's and the model's scores. The model says yes to both.

Inside, both are right. In what it says, Log B is wrong. To trace the information we ran Log B but replaced the model's internal state at one group of positions and one layer with Log A's (shown in the following picture), then measured how far the probe's reading moved toward Log A (shown in the Table):

Two grids with the same rows and columns, Log A, which ends at 63, above Log B, which ends at 45. Rows are layers 26, 24, 22, 20, 18, 14 and 2. Columns are the number, the question and the answer position. An arrow copies Log A’s state at the answer position, layer 24, into the same cell of Log B. A probe reads Log B’s answer position at layer 26. Each Log B cell shows the share of the probe’s reading that moved toward Log A when only that cell was swapped. Layer 2, number 95%, question 0%, answer position 0%. Layer 14, number 75%, question 2%, answer position 0%. Layer 18, number 65%, question 18%, answer position 2%. Layer 20, number 41%, question 45%, answer position 1%. Layer 22, number 9%, question 69%, answer position 18%. Layer 24, number 2%, question 4%, answer position 91%.

Figure 7. Swap one cell of the model’s notes from Log A into Log B, repeated for every layer and three spots.
Table 6. In Qwen3-8B, how far swapping in Log A's state at one group of positions and one layer moves the probe's reading toward Log A. Bold marks the largest share at each layer.
LayerSwap the numberSwap the questionSwap the answer position
295%0%0%
1475%2%0%
1865%18%2%
2041%45%1%
229%69%18%
242%4%91%
Table 6. In Qwen3-8B, how far swapping in Log A's state at one group of positions and one layer moves the probe's reading toward Log A. Bold marks the largest share at each layer.

The difference starts on the digits of the last value (95% at layer 2), moves to the question's words around layers 18–22 (69% at layer 22), and from layer 24 sits at the answer position: swapping in Log A's state there alone recovers 91% of the probe's reading, while swapping the number or the question at that layer recovers almost nothing (2% and 4%). By layer 24, nearly everything the model has worked out about whether the setting changed is gathered in the one spot where it produces its answer word. The pattern is stable across the 48 pairs (95% intervals: 87–98%, 63–77% and 88–95%) and repeats in a second, smaller Qwen model, Qwen3-4B (98%, 70% and 100%).

So the knowledge reaches exactly where the answer is produced, and still doesn’t become the word. One reason is geometric: the direction the probe reads and the direction that decides yes or no are almost at right angles (a cosine of 0.02, about 89°), so the yes/no read-out barely uses the direction where the knowledge sits.

A warning light, before any fix

If the answer is inside, it can flag the model's own mistakes. For each yes/no answer the model gave, we computed a suspicion score, how likely that answer is to be wrong, in three ways:

  • The model's own confidence: 100% minus how sure the model is of its answer. It needs no look inside, but it misses answers the model gives confidently and wrongly.
  • An error probe: a second probe on the same activations, trained to predict whether the model's answer is wrong.
  • The truth probe against the answer: how far the truth probe's reading (changed or not) is from what the model said. A big gap means one of them is wrong, and it is usually the model.

Each light is scored by how well it ranks wrong answers above right ones, not by how confident anything is. A score of 1.0 means every wrong answer gets a higher suspicion than every right answer; 0.5 means that, for a random wrong-right pair, the wrong one is higher only half the time, which is a coin flip.

Table 7. How well each warning light ranks the model's wrong answers above its right ones, from 0.5, a coin flip, to 1.0.
Task and modelWarning light from the model's confidenceFrom the error probeFrom truth probe vs answer
Rate of change, Qwen3-4B0.630.960.96
Rate of change, Qwen3-8B0.520.910.91
Topology, Qwen3-4B0.720.690.79
Topology, Qwen3-8B0.740.670.75
Table 7. How well each warning light ranks the model's wrong answers above its right ones, from 0.5, a coin flip, to 1.0.

Where the model knows (rate of change), the probes flag its mistakes well: of the 20% of answers they flag as most suspicious, 98–100% are truly wrong, against 71–76% for a random 20%, since that share of all answers is wrong anyway. Picking the 20% the model itself is least sure of does no better than random: its doubts don't point to its mistakes, because it is about as sure of its wrong answers as of its right ones (Qwen3-4B: a median of 97% sure when wrong, 99% when right). Where it can't (topology), the probes are no better than its confidence. Each light works only on the task it was trained on (0.52–0.61 on the other).

Fixing each failure

Steering shifts every answer the same way. Steering adds the same arrow to every question's activations at the answer position, at layer 28, a few layers after the knowledge reaches it (results were the same at layers 24 and 32):

h′ = h + α·σ·v̂

where v̂ is the direction of the push, scaled to length 1; σ is how much questions naturally vary along that direction; and α is the push size in those units.

We tried three directions, each pushed two ways: far beyond anything natural (11 to 13 times the gap between changed and unchanged settings), and all by the same raw amount at layers 24, 28 and 32:

  • The probe's own direction: nothing happens, even at α = 16. At the same raw size, it moves 3% of answers.
  • The average difference from unchanged to changed settings: every answer flips at α = 8. At the same raw size, it moves 93% of answers.
  • A random direction, as a control: almost nothing happens. At the same raw size, it moves 2–4% of answers.

Here is why (illustrated below). Along the probe's direction, questions vary little (σ = 0.70), yet changed and unchanged settings separate cleanly. Along the direction the answer depends on, questions vary a lot (σ = 35.4), and the two groups overlap.

Truly changed settingUnchanged, before the pushUnchanged, after the push

Two cartoon scatter plots of questions in two of Qwen3-8B’s internal numbers at layer 28. The vertical axis is the probe’s direction, along which questions vary little (sigma 0.70). The horizontal axis is the direction the answer depends on, along which they vary a lot (sigma 35.4). The two are nearly at right angles, cosine 0.02. Unchanged settings sit below the probe’s line and truly changed ones above it, and the two groups overlap left to right. Right of the answer’s line the model answers changed. Push 1, a push along the probe’s direction at alpha 16 moves the unchanged settings straight up across the probe’s line, and the model still answers unchanged. Push 2, a push along the average difference at alpha 8 moves them up and to the right across the answer’s line, and the answer flips to changed.

Figure 8. A cartoon in two of Qwen3-8B’s 4,096 internal numbers at layer 28. Each dot is one question (red for unchanged, teal for truly changed). A push adds the same arrow to every unchanged question (faded before the push, outlined after). Push 1, along the probe’s direction, crosses the probe’s line but not the answer’s, so the answer stays “unchanged”. Push 2, along the average difference, crosses the answer’s line, so every answer flips to “changed”.

The probe's direction and the direction the answer reads (the output layer's "yes" minus "no") are nearly at right angles, with a cosine of 0.02. So a push along the probe's direction changes what the probe reads but leaves the answer untouched: in the figure, it crosses the probe's line but never the answer's. Pushing along the average difference does move the answer, but it moves every question the same way. Pushed 4σ toward "not changed" (α = −4), the model's average probability of "changed" drops to 15% for truly changed settings and 12% for changed-back ones, so it still can't tell them apart. Steering changes how often the model says "changed", not whether it is right.

For "knows but doesn't say": retrain a few middle layers. The probe says the answer is already computed, and the swap experiment says it reaches the answer position by layer 24 but does not become the output word. So the repair should connect the two before that hand-off, not teach the task again. We retrained one block of four layers of Qwen3-8B at a time, everything else frozen: small enough to show where in the model a fix works, and the same size at every depth so the blocks compare fairly. Each block was trained on 6,400 questions from fresh logs, three runs per block:

Table 8. Held-out accuracy of Qwen3-8B after retraining one block of four layers with everything else frozen, in three runs per block.
Block retrainedHeld-out accuracy, three runs
None (original)24%
Layers 4–794%, 92%, 94%
Layers 14–1799%, 96%, 99%
Layers 20–2394%, 94%, 97%
Layers 24–2757%, 51%, 77% (75% with 3× training)
Layers 32–3545%, 47%, 60% (54% with 3×, 52% with a 10× learning rate)
Table 8. Held-out accuracy of Qwen3-8B after retraining one block of four layers with everything else frozen, in three runs per block.

Every block before layer 24 fixes it reliably; later blocks learn it slowly or barely, consistent with the hand-off we saw in the swap experiment. The layer 14–17 fix generalizes: 98% on harder logs, 95% on a reworded question, 93% and 97% on logs written as JSON events or plain sentences that it never saw in training (the base model scores 23–25% there), and its loss on 300 general chat replies does not get worse (2.013 to 1.987). The fix is spread out but small: no single layer or attention head group carries it, the MLPs carry most of it, each weight matrix moves by only 0.2–0.3% of its size, and keeping the top 256 directions of each weight change keeps 94%.

For "can't": thinking, if the model is big enough. Here nothing is inside to connect, so the fix has to add computation. A single forward pass, one trip through the model's layers to produce one word, can work out only a few hops. In the larger Qwen models, the answer is partly there inside at 2–3 hops (a probe gets 63–78% of failing services right, the model only 22–48%) and fades beyond that. Thinking lets the model write the chain out one hop at a time, each written step feeding the next pass. With thinking on, Qwen3-8B rises from 68% to 91% on 1,937 topology questions (answers that run out of room count as wrong).

But every step has to be right. On harder graphs with chains up to 22 hops, at 13+ hops Qwen3-8B with thinking reaches only 6%, even while writing thousands of reasoning tokens; Qwen3-235B-Thinking reaches 86% (99% at one hop). The large model fits a small error per hop adding up along the chain (0.99¹³ ≈ 88%). The small model falls much further than that, suggesting its steps get less reliable as the chain, and its own notes, grow longer. Thinking removes the one-pass limit, but only for a model whose steps are reliable.

Bar chart of accuracy with thinking on, on graphs with chains up to 22 hops. Qwen3-8B, 13 or more hops, 6%. Qwen3-235B-Thinking, 13 or more hops, 86%. Qwen3-235B-Thinking, one hop, 99%.

Figure 9. With thinking on, Qwen3-8B reaches 6% at 13 or more hops and Qwen3-235B-Thinking 86%, close to the 88% that a 1% error at each of 13 hops would give.

What about Claude?

Does Claude fail the same way? We can't look inside it, but we can ask it the same 640 yes/no settings questions. Claude gets them right, Haiku 96% and Sonnet 99.5%, because both reason before answering: Haiku in visible text, Sonnet in hidden thinking on most calls, even with thinking set to off. Forced to answer in one word, Haiku drops to 80% and fails the other way from Qwen, missing 71% of real changes; Sonnet stays at 99.5%. So the failure we dissected belongs to instant one-word answers from open models, and whether Haiku's one-word mistakes are "knows but doesn't say" or "can't" is something only its internals could tell.

What we learned

  • Where models break. Long chains, concurrency, churn and look-back rules break Haiku until it thinks. In two stress tests that mean tracking hundreds of facts exactly, counting failing services breaks every Claude model, Opus included, and following 1,024 edits by 128 writers breaks Haiku and Sonnet and pulls Opus down to 80%.
  • Looking inside. On settings, a probe reads the right answer from the activations: the model knows but doesn't say, and the probe can flag its mistakes. On long chains, no probe finds it: the model can't.
  • Saying it right. Steering shifts every answer the same way. Retraining a few middle layers fixes "knows but doesn't say"; thinking fixes "can't" when the model is big enough.
  • Diagnose before fixing. The same wrong answer can mean "never computed" or "computed but not used". One probe and one control tell them apart, and the diagnosis predicted which fix worked.

Limitations

The tasks are synthetic text, with one generator per task and 30 tasks per point in the Claude curves, so each point carries a 95% margin of error of about ±15 points (up to ±18 near 50%). Claude's thinking could not be fully switched off through our interface: Sonnet thought on about half of its "off" calls. The internal analysis uses Qwen and OLMo, not Claude. Patching, steering and retraining are on one model, Qwen3-8B, with three runs per retraining block. We did not vary the block size, so four layers is a choice, not a minimum.

What's next

  • Does scale close the gap? Bigger models narrow it (72B: model 0.92, probe 0.99). We want to know whether scale fixes the read-out, lets one pass resolve more hops, or both.
  • Do the probes work on real systems? Run them and the warning light on real configuration diffs and incident logs.
  • How does the model compute "changed", and why does the answer ignore it? Find the parts of the model that compare the current value with the snapshot, and what the yes/no answer reads instead.
  • Is the long-chain failure in reading the graph or in following it? Check whether the model even holds each "A calls B" link, before blaming the chaining.

Appendix: setup details

Table 9. Setup details for the experiments in this post.
ItemSetting
Claude modelsHaiku 4.5, Sonnet 5.5, Opus 5.5 (stress tests), via a command-line interface, thinking off and on
Open modelsQwen3-4B-Instruct, Qwen3-8B, Qwen3.6-27B, Qwen2.5-72B-Instruct, OLMo-2 7B Instruct; thinking off
ProbeStandardized activations, L2 logistic regression (C = 0.05); layer chosen by 5-fold cross-validation on 210 training logs; scored on 90 held-out logs
Rate-of-change data300 logs × 16 settings = 4,800 questions per model
Topology data300 dependency lists of 8 to 128 services, about 1,900 questions per model
Patching48 log pairs, Qwen3-8B (and Qwen3-4B); probe read at layer 26; recovery = (after swap − Log B) ÷ (Log A − Log B)
SteeringQwen3-8B, answer position, layer 28 (24 and 32 for matched size); 640 questions from 40 fresh logs
RetrainingAdamW, learning rate 1e-5, batch 8, 800 steps, class-weighted loss on the yes-minus-no score; three runs per block
Table 9. Setup details for the experiments in this post.
More from the blog
All posts →