The model's activations know more than it says
I showed Llama 24 held-out biographies and asked how likely each factual claim was to be correct. It gave 650 of 742 claims a probability of exactly 1.0. A linear probe that reads Llama's own layer-19 activations ranks the same claims far better, at 0.802. The stronger comparisons are closer: P(True), which asks Llama about one claim at a time, reaches 0.735, and a lookup of how famous each subject is reaches 0.740.
Scores below are AUROC: the chance that a randomly chosen correct claim is scored above a randomly chosen incorrect one. 0.5 is a coin flip; 1.0 is perfect. Brackets are 95% bootstrap intervals; each says whether it resamples claims or whole biographies.
All four scores rank the same 742 claims from 24 biographies held out from probe training; intervals resample claims. With only 24 biographies, resampling whole biographies gives the probe a wider [0.748, 0.848]. Probe + P(True) is exploratory.
Read a biography
Each biography was written by ChatGPT and split into atomic claims, which FActScore's annotators marked as supported or not. Pick a signal and flag the claims it trusts least. Highlighted sentences are the ones it flags; a dotted underline marks sentences that contain an incorrect claim.
Click a sentence to find its claims. The probe reads whole sentences, so claims from one sentence share its score. Llama-as-verifier saw the whole biography and the claims in chunks of up to 20; P(True) saw one claim at a time with the subject named.
Ranking every claim
Flag the claims a signal trusts least and count how many of the 233 incorrect claims you catch. A signal that knows nothing catches errors only in proportion to how many claims you flag (the diagonal). The marker follows the slider above.
Errors caught as more claims are flagged
Ties are drawn as straight segments, the average over random tie-breaking. Llama-as-verifier's long flat-to-diagonal run is the 650 claims it scored exactly 1.0.
Within one biography
Pooled over all claims, popularity alone does well: a subject's Wikipedia pageviews reach 0.740 [0.701, 0.780], and with whole biographies resampled no clear advantage of the probe over them is detected (+0.062 [−0.017, 0.155]). Comparing only claims about the same person holds popularity fixed.
Within-biography AUROC on the 24 test biographies, with whole biographies resampled. The probe leads Llama-as-verifier by +0.155 [0.057, 0.244]. Sentence position scores earlier sentences as more likely correct; it needs no model and, unlike popularity, varies within a biography.
All 24 test biographies
Within-biography AUROC for each one, sorted by the probe. Some biographies have few incorrect claims, so single-biography scores are noisy. Click a name to read it.
Two signals beat one
P(True) asks Llama about one claim at a time and reads how strongly its first answer token favours “True” over “False”. It is a much stronger baseline than the stated probability, at 0.735 [0.696, 0.772]. The probe leads it across the test set (+0.067 [0.003, 0.124], biographies resampled), but within biographies no clear advantage is detected (+0.029 [−0.009, 0.068], all 157). The two disagree often (correlation 0.35), and a linear stack with two weights beats both.
Within biographies, all 157 (cross-fitted, biographies resampled)
The stack gains +0.055 [0.039, 0.072] over the probe and +0.085 [0.056, 0.113] over P(True). Facts from one sentence share the probe's score, so part of the gain over the probe is the stack breaking those ties; leaving out same-sentence pairs, the gain is +0.042 [0.025, 0.059].
Stack weights were fitted within biographies on the other 133 biographies: 0.0729 per unit of probe logit and 0.0283 per unit of P(True) logit difference.
Every test claim, by both signals
The vertical axis reads P(True) in logits: Llama's logit for “True” minus its logit for “False”. Incorrect claims (filled) gather toward the lower left. Each dashed line joins claims with the same stack score, which rises toward the upper right. Hover a point to read its claim.
Where the signal lives
The probe reads one direction in Llama's layer-19 residual stream. Everything written to that stream comes from an attention head, an MLP neuron or the token embedding, so the probe's score splits exactly into a direct contribution from each of 286,720 neurons, 640 heads and the embedding. I measured each component's share of the probe's within-biography separation between correct and incorrect claims, after removing what position and year mentions explain. These are direct, signed shares under this decomposition, not shares of Llama's factual knowledge or of all the computation behind the signal.
Heads carry 79% of the direct separation; neurons nearly cancel out
Bars show the summed contributions that push correct claims above incorrect ones (right) and the reverse (left); the dot is the net. Neurons push 18.23 one way and 18.06 the other, leaving a net 0.173, or 21% of the total 0.814. The embedding's share is 0.01%.
640 attention heads
Layers 0–19 down, heads 0–31 across. Pink heads rank correct claims above incorrect ones, net; blue heads do the reverse. The ten outlined heads carry 33% of the separation between them; the strongest alone carries 5.0%.
Where the top ten heads look
Average attention from a sentence's tokens, split by what they attend to. Descriptive.
286,720 MLP neurons, one pixel each
Each square is one layer's 14,336 neurons. Pink neurons rank correct claims above incorrect ones, net; blue neurons do the reverse; colour strength shows how a neuron's mass ranks among all of them. Slide to light up, in amber, the neurons with the most positive mass, and see how little of it any small set holds.
The 50 neurons with the most positive mass, the positive carriers rated below, hold 1.25% of it. Half of it takes 16,307 neurons and 80% takes 46,127. Across the five folds, the top-50 sets overlap with Jaccard 0.53 on average.
Knocking components out
If a set of components carries the signal, replacing their output with its average should hurt the probe more than doing the same to random components from the same layers (neurons are also matched on how much their contribution varies; heads only on layer). Scores here are within-biography AUROCs of the position-residualized probe, which starts at 0.644 (the fp32 counterpart of the position-residualized 0.645). In the preregistered test, knocking out each fold's 200 neurons with the most mass, chosen on the other four folds, costs 0.032 of within-biography AUROC, against 0.006 for matched random neurons. The targeted intervention had a measurable effect, 0.026 [0.013, 0.038] beyond random, but it did not meet the 0.03 effect-size criterion set before the test.
Exploratory follow-ups, each planned before its data: E1 knocked out the top 10 heads, and E2 extended both curves. Cost grows steadily with set size, from 0.019 for the top 10 heads to 0.213 [0.179, 0.250] for the top 160 heads and 0.219 [0.180, 0.259] for the top 5,000 neurons. Random sets, averaged over three per size, never cost more than 0.010. At 20,000 neurons they slightly raise the score (−0.019), so top − random there overstates the top set's cost.
At the largest sizes the residualized score falls below chance (0.425, 0.376 and 0.431 after 5,000 neurons, 20,000 neurons and 160 heads). On the raw probe logit, which starts at 0.689, the same knockouts leave 0.485, 0.406 and 0.534, so only the 20,000-neuron set is clearly below chance on both scales. Why is untested: the selection took only components with positive mass, so those pushing the wrong way stay in place, and the residualized score subtracts fixed position slopes. Read these points as showing dependence, not measuring it.
Plotted random points average three sets. The preregistered test and E1 used five, giving top − random differences of 0.026 and 0.013 where these three-set points give 0.025 and 0.010. Intervals resample biographies.
Every knockout as a table
Neuron browser
An LLM rater saw 200 neurons in shuffled order, without knowing which were which: the 100 carriers (the 50 with the most positive and the 50 with the most negative mass) and 100 random neurons from the same layers whose contributions to the probe score vary about as much, drawn from outside the top 1,000 by |mass|. For each it wrote a description and judged whether the description fits, and whether the neuron tracks knowledge (epistemic), a confound such as topic or format, or a mix.
Legible, and read like random neurons
98 of 100 carriers and 95 of 100 random neurons got a description that fits. Weighted by mass, the coherent carriers are 90% confound, 9% mixed and 1.2% epistemic: Primary A reads CONFOUND-LED. Confound features are no more common among carriers than among random neurons (87/98 against 91/95, Fisher p = 0.10). The preregistered rubric counts a claim type or topic (awards, education, family) as a confound, so both groups come out confound-heavy: the label reflects the rubric as much as the carriers, and the comparison with random neurons is the informative one.
Primary B asks whether the top carriers are as legible as Cammarata's essay “Neural Networks Are Already Clean” predicts. It reads ESSAY-CONSISTENT for the top carriers of a dense probe, on a test near its ceiling: 2% of carriers got a description that did not fit their sentences, against 5% of random neurons. But the 50 positive carriers hold only 1.25% of the positive neuron mass, and neurons carry 21% of the direct separation.
No clear advantage for carriers' descriptions
A second model (gpt-6.1-sol) used each description to predict the neuron's activations on held-out sentences. No clear advantage was detected for carriers' descriptions: Spearman 0.33 against 0.27, a difference of 0.062 [−0.014, 0.137] with neuron pairs resampled. The interval rules out a carrier disadvantage beyond 0.014, but not an advantage of up to 0.14. The control that would set a chance level was not run. Both sides are LLM judgements: no human wrote or checked either.
The rater of record is an LLM (Claude Opus 5.5). A second pass of the same model agreed with κ 0.83 on fit and 0.76 on category, which shows consistency, not validity. That is the same circularity raised against the claim being tested. This version leaves out which neurons were carriers, so the packet can still be rated blind.
Mass is a neuron's net effect on ranking; activation is not. A neuron's effect on the probe score is its activation times its readout, so for 32 of the 50 positive carriers higher activation lowers the probe score, and their mass comes from being less active on correct claims. 8 carriers change readout sign across folds. This version leaves the per-neuron readout out.
Biography sentences are ChatGPT-written FActScore text and may contain false claims about real people. ✗ counts the claims annotators marked unsupported. Web-text contexts are unfiltered FineWeb excerpts and can include adult or offensive text.
What this does not show
- Popularity explains much of the pooled score. Pageviews alone reach 0.740, and with whole biographies resampled no clear advantage of the probe over them is detected (+0.062 [−0.017, 0.155]). The within-biography comparisons are the cleaner test.
- Position is a real confound. Later sentences in a biography are wrong more often. Across all 157 biographies, position alone reaches 0.654 within biographies against 0.689 for the probe. Removing the probe's dependence on position and year mentions leaves 0.645 [0.614, 0.675] (pass bar 0.60, set before the decomposition: PASS), the claim-level signal I can defend.
- Sentence pooling. The probe averages activations over a claim's source sentence, so claims in one sentence share a score.
- Llama-as-verifier is not self-report. ChatGPT wrote these biographies; Llama only judged them. P(True) is the stronger read of Llama's own confidence.
- The decomposition counts direct effects only. Each component is credited with what it writes straight into the layer-19 stream. Influence that passes through later components is credited to those components.
- The preregistered knockout had a measurable effect but missed its effect-size criterion (0.026 against 0.03). The probe + P(True) stack, the head knockout (E1), the dose-response curves (E2) and the simulation scoring (E3) are exploratory: each was chosen after related results were known, then fixed in a dated plan before its own data existed.
- Legibility was judged by an LLM. Nearly every neuron looks coherent to it, carriers and random alike, so the rating has little room to tell them apart. Its rubric files topics under confound, which shapes the Primary A label.
- Small test split. 24 biographies. The probe scored 0.728 on validation and 0.802 on test; resampling whole test biographies, the interval is [0.748, 0.848].
- One model, one layer, one dataset. Llama 3.1 8B Instruct, layer 19, 157 ChatGPT biographies and 4,886 human-labelled claims.
Sources
Every number on this page is read by scripts/c01/build_explorer.py from the committed result files, or from the local, hash-checked artifacts named in the table. Those the paper prints are checked against it, signed, at the paper's precision. Charts are drawn from the per-claim and per-component data embedded in the page. Labels appear as the preregistered scripts recorded them: necessity_verdict NOT-NECESSARY means the 200-neuron knockout's 0.026 beyond random missed the 0.03 criterion, not that those neurons had no effect.