Agents traced a lookup pathway in Qwen3

[ 2026-10-23 ] // gpt-6-luna · Qwen3-0.6B · Qwen3-1.7B // 15 MIN READ

Tell a language model that Rowan is red and Lumen is blue, then ask what color Rowan is. To answer, it has to look up the value that belongs to the named item. I wanted to see how a model does that, and whether an explanation found in one model could predict what happens inside a larger one.

A small task with a revealing change

The task is deliberately simple:

Rowan is red.
Lumen is blue.
What color is Rowan?

Now swap the names in the facts:

Lumen is red.
Rowan is blue.
What color is Rowan?

The question is identical. The colors appear in the same order. Only the relationship between names and colors has changed, and the answer must change with it. (Rowan and Lumen are illustrative. From study 4 on, the keys were other single-token words, drawn fresh for each study without reference to the model’s answers.)

A signal that only carries the name “Rowan” would be the same in both cases. A signal that incorporates which record belongs to Rowan must reflect the changed context. To tell them apart, the experiment runs the model on both versions, then transplants selected internal signals from one run into the other and measures what changes. This intervention is called activation patching.

The outcome measured throughout is the model’s answer preference: how strongly it favors one of the two candidate answers, such as red over blue, as the next word. It is measured in logits, the raw scores the model assigns to possible next words. Free-form generation was not tested. A second kind of swap asks about the other name with the facts unchanged; only the name swap above rules out simply copying the name from the question.

One especially informative experiment changed the signal the copying heads use to choose where to read, while keeping their other lookup inputs fixed. In the confirmation of the last two stages, the three copying heads’ average attention shifted toward the record selected by the transplanted signal by a median of 14 to 21 percentage points, depending on the comparison, and their contribution to the answer moved with it. In the name swap, where the question stays the same, that signal still changed in a way that affected which record was selected, so it was carrying information about the relationship between the question and the facts.

Three stages: identify, relay, select

Each of these models has 448 attention heads, 16 in each of 28 layers. The investigation found twelve that carry a linked lookup pathway, in three groups with different roles:

1. Identify the requested record. Six heads in layers 11 to 14, working over the question and the text that follows it, influence a signal that depends on which record answers the question.
2. Relay it to the answer position. Three heads in layers 15 to 18 read that information from the question region and carry it to the position where the model is about to produce its answer.
3. Select the source and copy its value. Three later heads, in layers 19 to 22, use the relayed signal to direct their attention to one record’s value and copy that value into the answer.

The experiments let a change travel only through the proposed route while holding other inputs fixed, which separates an effect through the tested components from a direct effect or one that bypasses them.

In the smaller model, the full three-stage intervention moved the answer preference in the predicted direction on all 192 held-out prompts (128 in the main set and 64 with new answer colors), and on the 64 calibration prompts used to select the heads. “Predicted direction” means the preference moved toward the answer associated with the transplanted signal. That study passed all 63 of its mechanism criteria but failed its overall verdict: its registered prerequisite was that the unmodified model answer at least 103 of the 128 main held-out prompts correctly, and it answered 101. The mechanism results stand as measured, but the pathway was not fully confirmed in the smaller model.

Separate tests isolated the copying step. In the confirmation of the last two stages, changing the value information at the color positions moved the answer preference by a median of 4.14 logits, against a median absolute 0.02 when the same change was made elsewhere. That confirmation also passed on fresh color words, green and yellow, without reselecting any heads.

An address is a useful analogy for the routing signal: it helps determine where to read before the value is copied. That is a functional interpretation. The causal experiments establish the signal’s role in selecting a source; they do not decode a literal address stored in the network.

The smaller model supplied a prediction for the larger one

The two models were Qwen3-0.6B and Qwen3-1.7B: roughly 600 million and 1.7 billion parameters, the numerical settings learned during training. Their head layouts match, so a head’s location in one model names the same location in the other.

In Qwen3-1.7B, a separate search on calibration prompts first located the last two stages. It recovered the same three copying heads as the smaller model, and two of its three relay heads matched; the third was 18:7 in place of 15:4. That study passed all 108 of its registered criteria, 36 of them on the calibration prompts used for the search.

Then came the prediction. The six identifying heads had been found in the smaller model, and their locations were fixed before the transfer test in the larger model, with no new search for them. The study reused the smaller model’s 63 mechanism criteria unchanged and asked whether these six locations would feed the larger model’s relay in the expected way.

They did. Interventions starting at the six heads moved the larger model’s answer preference in the predicted direction on 92.2% to 100% of held-out prompts, across both kinds of swap and both held-out prompt sets. The study passed every registered gate, and the unmodified larger model answered 127 of the 128 main held-out prompts correctly.

The 92.2–100% figure counts direction only: it is not answer accuracy, the effect sizes varied from the smaller model’s, and it does not show that the pathway is a complete account of lookup. A separate, ungated check that carried all twelve original locations unchanged into the larger model fell short of the main threshold on one measure.

That connects to a question from my recent work on reusing proofs of OpenSSL code: can a result become something the next investigation starts with? Here, a search in one model supplied concrete targets for an experiment in another, removing the need to repeat that particular search.

A model can retrieve the right value and still answer incorrectly

On the first study’s 128 held-out two-record prompts, the full model made thirteen errors. A readout of one copying head, 19:5, favored the correct value on all thirteen. In a separate test using “gold” and “silver,” the same head favored the correct value on all nine of the model’s calibration errors in that test. Neither study passed its own overall test: the first missed its accuracy prerequisite by one prompt, and the gold/silver follow-up failed a calibration prerequisite. The readouts are descriptive findings from those runs.

Interventions showed that this head also pushed the output toward the correct answer: its signal was participating in the computation, and later processing could outweigh it. That suggested a possible repair: strengthen the copying heads’ contribution.

On four-record prompts, the copying heads had different preferences for where to look: 19:5 leaned toward the last record, while the other two leaned away from it. Simply amplifying the heads also amplified values retrieved from the wrong position. The first amplification attempt raised the median margin on the prompts used to choose it, yet made 15 of their 17 existing errors worse.

The later repair accounted for those position preferences. It subtracted part of each head’s habitual lean toward particular records before the head mixed in values, and amplified head 22:9. Its settings were chosen on calibration prompts and then frozen. On the 192 held-out four-record prompts, it fixed six errors and introduced one new error, a net gain of five; naive amplification of all three heads fixed ten and broke seven on the same prompts. The repair still failed its registered test. On the main held-out set it did not improve the margins of the recency errors it targeted, and on fresh colors its net gain only tied naive amplification. A separate repair on the gold/silver task fixed two errors and broke two.

The value of this small pilot was the link between explanation and intervention: knowing how the heads divided the task suggested a more targeted repair than amplifying every head, with fewer new errors on both four-record sets.

My recent post on what Llama’s internal signals reveal about factual accuracy asked what activations reveal about whether an answer is correct. This project traces how an answer is selected, relayed, and sometimes outweighed before it reaches the output.

What the agent harness enabled

By agent harness, I mean the software workflow that gives the agents tools to build controlled prompts and change internal computations, runs the experiments, checks the outputs against criteria fixed in advance, and keeps a record of everything tested.

The agents, Codex CLI calls running gpt-6-luna, proposed explanations and reviewed results. I directed the campaign: I specified or approved changes to the designs, word lists, record formats and pass criteria before the runs that used them, and authorized the compute budget for the transfer study. The agents could not turn a failed test into a pass: frozen, executable tests decided which explanations survived, and no finished study was regraded.

Test competing explanations. One plausible account held that the names were stamped onto the colors in the facts, so the copying heads could match on them directly. A registered test found that route too weak to meet its bar; the effect of swapping names ran mainly through the signal that tells the copying heads where to look. Planted control models with deliberately simple behavior were run before the Qwen3 tests to check that the tests could tell the explanations apart; from study 5 on, they included one that only copies the literal name from the question.

Follow a chain of interventions. Finding the heads that copied the value raised the question of what told them where to read. Tracing that signal backward led to the relay heads, and tracing the relay’s input led to the six heads at the question.

Make predictions before measuring. Before each confirmation run, the harness froze the selected heads, prompts, interventions and pass criteria, and recorded file fingerprints. Heads were chosen on calibration prompts and tested on held-out prompts. The transfer study is the clearest case: name six locations first, then measure what happens there in the other model.

Keep failures in the record. The first amplification attempt made answers worse, and that pointed to the position preferences the later repair used. A version with structured, JSON-like records failed both its tracks, and its measurements pointed the search back toward the question. The appendix lists every study and its verdict.

Several checks stood behind each result. Each confirmation ran twice in fresh processes and had to reproduce its saved numbers exactly, and a separate NumPy script recomputed every gate from the saved arrays. In the later studies, the AI review cited numbers by reference and code filled in the audited values, so it could not misquote them. These are internal checks, not outside review.

All of this ran on one 24 GB M4 Pro MacBook Pro. Broader uses, such as starting new investigations from a tested map of a model, monitoring deployed models, or repairing errors automatically, are future possibilities; this campaign shows one prospective reuse and one small repair that did not pass.

Where this fits

Prakash and colleagues traced entity-tracking circuits in Llama-7B, multi-stage groups of heads that connect a question to the correct value. Gur-Arieh, Geva and Geiger’s work on mixing retrieval mechanisms shows that models combine different strategies for retrieving bound entities from context.

What this project adds is a causal account in two Qwen3 sizes tested against registered criteria, a prospective test of locations carried from one size to the other, and a record that keeps failed studies beside passing ones.

Limits

The most consequential limit is scope. These are short, synthetic lookup prompts with two or four records, in two related models. “Answer preference” and accuracy here mean the model’s preference between two specified answer words, such as red and blue, at the next token. Free-form generation was not tested. Longer contexts, other model families and more complicated questions were not tested either, and the one attempt at structured, JSON-like formats failed.

• The confirmation of the last two stages passed after three of its thresholds were revised before the run, using earlier outcomes: accuracy from 85% to 80%, and two attention-shift bars. Earlier failed studies were not regraded. The later three-stage studies kept the 80% accuracy bar.
• Heads were selected on calibration prompts, so calibration results are selection evidence, not confirmation. The selected heads are not shown to be unique or exclusive, and this is not a complete account of how the model does lookup.
• Matching locations across the two sizes do not prove the heads are the same component. The pathway that passed in the larger model uses 18:7 in place of 15:4.
• The two fresh runs per confirmation show the numbers repeat exactly; they are not extra samples. The separate audits reuse some of the same NumPy arithmetic as the production code. No one outside the project has reviewed the work.

Next experiment

The scope limit has a direct test: carry the fixed heads, unchanged, to longer prompts and structured formats, and require the same direction and size criteria to pass with no new search. Another model family would need its own search first, because its layout differs.

One question remains open: what does the routing signal represent? It may encode position, aspects of the requested record, or a combination. Moving records around independently of their names and values can help distinguish those possibilities.

For the harness, the next question is whether giving agents an existing causal map lets them solve a new research task with less search. Answering it requires comparing investigations that start with such a map against ones without it.

Appendix: every study and its verdict

An earlier broad sweep over real text tested 36 hypotheses and produced no survivor, which is why the campaign moved to this controlled task. The table lists the controlled studies in order, with the verdict set by each study’s own criteria. Post-hoc diagnostics of finished studies are described in the results ledger and are not counted as confirmations.

Reproducibility