What Llama’s internal signals reveal about factual accuracy

[ 2026-10-18 · updated 2026-10-23 ] // Claude Opus 5.5 · Llama 3.1 8B // 20 MIN READ

This is the second stage of a project that began with Testing internal signals for factual accuracy in legal answers and biographies. There, a probe, a small readout trained on Llama 3.1 8B’s internal activity, separated correct from incorrect biography claims far better than the model’s stated confidence, a weak comparison. Such a readout could cheaply flag likely false claims in long answers, but only if it carries information simpler signals don’t. Here I test it against stronger baselines, check whether it only tracks a subject’s fame or a claim’s position, and ask which components write the signal it reads.

Setup and baselines

FActScore (Min et al., 2023) has 157 ChatGPT-written biographies split into 4,886 atomic facts, each marked supported or not by human annotators; those labels are the ground truth for every score here. For each fact, I average Llama 3.1 8B Instruct’s layer-19 residual stream (the running state each layer reads from and adds to) over the fact’s source sentence. The probe is an L2 logistic regression fitted on 133 biographies and scored once on 24 held-out ones: 742 facts, 233 of them wrong. I compare it with four baselines, two from Llama itself:

• Stated probability. Llama sees the biography and up to 20 facts at a time and writes a probability that each is correct. It did not write the biographies, so this is Llama verifying ChatGPT’s claims (the explorer’s Llama-as-verifier).
• P(True). A separate way of asking Llama whether a claim is true: I show it one fact, with the subject named, and read how strongly its first answer token favours “True” over “False” (the logit difference).
• Popularity and position. The subject’s Wikipedia pageviews, and the fact’s sentence position, scoring earlier sentences as safer.

Scores are AUROC, the chance that a random correct claim outscores a random incorrect one (0.5 is chance). Within-biography AUROC compares only claims about the same person. Brackets are 95% bootstrap intervals over claims or whole biographies, as stated.

Han et al. (2025) also probed Llama hidden states for claim correctness, testing on FActScore against verbalized confidence and P(True). Their Llama 3.1 8B probe, trained on LongFact, reaches AUROC 0.698 on ChatGPT claims from the same human-annotated FActScore release; my 0.802 comes from a probe trained on FActScore itself, so it is not an improvement on theirs. This study adds paired comparisons with stated confidence and popularity, within-biography controls, the probe–P(True) combination and a decomposition with knockouts.

Stronger baselines narrow the gap

The stated probability fails by saturating: 650 of the 742 claims get exactly 1.0 and 39 exactly 0.0. The probe leads it by +0.262 [0.180, 0.332] with whole biographies resampled, and by +0.155 [0.057, 0.244] within biographies. Over P(True), the stronger read of Llama’s judgement, its lead is +0.067 [0.003, 0.124] with biographies resampled, an interval that nearly reaches zero.

Popularity is the obvious confound. Facts about obscure people are wrong more often, so a signal could rank claims by fame without reading them. Pageviews alone reach 0.740. The probe leads them by +0.062 [0.024, 0.097] with claims resampled, but popularity belongs to a whole biography, and with whole biographies resampled the interval is [−0.017, 0.155]: no clear advantage detected. Comparing claims within each of the 24 test biographies holds popularity fixed; there the probe’s lead over the stated probability holds, while over P(True), +0.068 [−0.049, 0.158], no clear advantage is detected.

Position is a confound too, and the signal survives it

Later sentences in a biography are wrong more often, and the probe’s score partly tracks that. Twenty-four test biographies are too few for this question, so I cross-fitted five probes, each scoring the fifth of all 157 biographies it never saw; the 24 test biographies now also help train the other folds’ probes. Position alone then reaches 0.654 within biographies, against 0.689 for the probe. Removing the probe’s linear dependence on position and year mentions, with slopes fitted on the other folds, leaves a residualized score that still reaches 0.645 [0.614, 0.675], above the 0.60 bar that gated the preregistered analysis below. This is the claim-level signal I can defend, and the one the decomposition takes apart.

The probe and P(True) carry partly different information (exploratory)

Over all 157 biographies the probe scores 0.689 within biographies and P(True) 0.660: no clear advantage for the probe is detected (+0.029 [−0.009, 0.068]). The two are only weakly correlated (0.35 on the test split), so I tested a linear stack with two weights fitted within biographies on the training folds.

Within biographies over all 157, the stack reaches 0.745 [0.719, 0.769]: +0.055 [0.039, 0.072] over the probe and +0.085 [0.056, 0.113] over P(True). Part of the gain is breaking ties among claims from one sentence, which share the probe’s score; without same-sentence pairs the gain over the probe is +0.042 [0.025, 0.059]. On the test split the stack reaches 0.834 against 0.802. I chose this test after seeing the single scores: it shows the signals are complementary on this data, but confirming the gain’s size needs biographies that played no part in choosing it.

Which components write the signal

I tested a claim from Cammarata’s essay “Neural Networks Are Already Clean”: that probe-like directions break down into a few readable neurons. Everything in Llama’s layer-19 residual stream was written by an attention head (a component that routes information between positions), an MLP neuron (one unit of a layer’s feed-forward block) or the token embedding, so the probe’s score splits exactly into direct contributions from 286,720 neurons, 640 heads and the embedding. I measured each one’s share of the residualized probe’s within-biography separation of correct from incorrect claims: a direct, signed share, crediting only what a component writes straight into layer 19 and netting out opposite contributions. It is not a share of what Llama knows or of the computation behind the signal.

Heads carry 79% of the direct separation and neurons 21%. The heads’ part is somewhat concentrated: the top head carries 5.0% and the top ten 33%. The neurons’ part is not: they push hard both ways and nearly cancel. The top 50 hold 1.25% of the positive neuron mass, half of it takes 16,307 neurons and 80% takes 46,127, and top-50 sets chosen in different folds overlap with a Jaccard index of 0.53 on average (neurons in both over neurons in either). These shares are preregistered descriptives, not tests.

Knocking components out

If a set of components carries the signal, replacing their outputs with their average (mean ablation) should hurt the probe more than doing the same to random components from the same layers. Selection and scoring were split by fold: for each fifth of the biographies, components were chosen by mass on the other four-fifths, knocked out while Llama read that fifth, and scored with that fold’s probe, though the probes that weight the other folds’ contributions were trained with this fifth included. Over all 157 biographies the residualized probe’s within-biography AUROC starts at 0.644 (the 0.645 above, recomputed in full precision).

The preregistered test knocked out each fold’s 200 neurons with the most mass. That cost 0.032 [0.016, 0.046] of AUROC, about a fifth of the residualized probe’s lift above chance and five times the 0.006 removed by matched random neurons. The targeted intervention had a measurable effect, 0.026 [0.013, 0.038] beyond random, but it did not meet the 0.03 effect-size criterion set before the test.

In exploratory follow-ups the cost grows steadily with set size, while random sets never cost more than 0.010. The largest sets push the residualized score below chance (0.425, 0.376 and 0.431 after 5,000 neurons, 20,000 neurons and 160 heads); on the raw score, which starts at 0.689, they leave 0.485, 0.406 and 0.534, so only the 20,000-neuron set is clearly below chance on both. Why is untested, so I read these points as showing dependence, not measuring it.

Neuron descriptions did not set the top neurons apart

To test readability, an LLM rater (Claude Opus 5.5, made rater of record by an amendment before any rating existed) described the 100 carriers, the 50 neurons with the most positive and 50 with the most negative mass, alongside 100 matched random neurons, shuffled and unlabelled. Plausible descriptions proved weak evidence of readability. These are LLM judgements; the claims’ correct and incorrect labels are FActScore’s human annotations.

• Carriers and random neurons read alike. Nearly all got a fitting description, so the test sits near its ceiling, and both came out mostly confound-led, partly because the rubric files topics such as awards as confounds.
• No clear advantage for carriers’ descriptions. In an exploratory simulation test, a second model used them to predict activations on unseen sentences; carriers’ Spearman correlation exceeded random neurons’ by 0.062 [−0.014, 0.137].
• Activation is not effect. The rater saw activation only, yet for 32 of the 50 positive carriers higher activation lowers the probe’s score.

Explore it

The Fact Probe Explorer holds the data behind this post, from per-claim scores on the 24 test biographies to every head and neuron, the knockout curves and the 200 rated neurons. It leaves out which neurons were carriers, with their layers, masses and readouts, so the packet can still be rated blind.

Who did what, and what checked it

I ran this stage with Claude Opus 5.5 as a coding agent; it co-authored every commit, from analysis code and GPU jobs to the preregistration and paper draft, and fresh Claude subagents did the ratings. The plan records two decisions as mine: an LLM as rater of record, and gpt-6.1-sol, from another model family, as simulator. The checks were mechanical: the plan and amendments A1–A3 predate the data they cover, scoring scripts stop on a failed hash or replication check, and an audit of all 40 rater transcripts found no access to the key. No human reviewed the ratings.

Interpretation and limits

A simple probe on Llama’s hidden states ranks human-labelled claims much better than Llama’s stated confidence and ahead of P(True) on the pooled test set, with no clear advantage detected over a popularity lookup. Within biographies it survives position controls and shows no clear advantage over P(True), though the two combined beat either alone (exploratory). What it reads is distributed: most of the direct contribution comes through attention heads, and I found no small, readable set of neurons that carries it.

The most consequential limitation is that no untouched biographies remain. The comparisons that hold popularity fixed rest on the 24 test biographies, or on a cross-fit over all 157 that includes them and was designed after their results were seen. The probe was also trained on pooled claims, so it can lean partly on popularity. Other limits:

• One model, layer and pooling choice, fixed in advance.
• The test split is small: the probe scored 0.728 on validation and 0.802 on test, [0.748, 0.848] with whole test biographies resampled.
• The decomposition counts direct effects only.

Next experiment

• A probe trained within biographies, so it cannot lean on popularity, and trained to ignore position, scored against P(True) and the stack on human-labelled biographies none of these analyses has touched.
• Blind human ratings of the same neuron packet.
• Sampled self-consistency and a semantic-entropy-probe baseline.

Methods details

Decomposition. Llama was re-run in fp32, capturing for layers 0–19 the inputs to each MLP down-projection and attention output projection and the token embeddings, mean-pooled over the probe’s sentence spans. These projections have no biases, so the pooled layer-19 residual is exactly their sum, and the components reproduce the probe’s score to within 0.002 logit. Each fact is scored with its own fold’s probe; a component’s mass is its mean true-minus-false difference over the 27,276 within-biography pairs.

Knockouts. Random neurons, here and in the ratings, come from the same layers, are matched on contribution variance and exclude the top 1,000 by absolute mass. Each knocked-out output is replaced at every answer token with its mean over all answer tokens of the 157 biographies.

Neuron ratings. The 100 carriers, unlike the knockout sets, were picked by mass over all 157 biographies, as the preregistration specified. The rater saw each neuron’s highest- and lowest-activating biography sentences and web-text contexts, wrote a description, judged its fit, and labelled the neuron epistemic (tracking knowledge), confound (such as topic or format) or mixed. A second pass of the same model agreed with chance-corrected κ 0.83 on fit and 0.76 on category: consistency, not validity. The simulation test had no chance-level control.

A data error. In one training biography a decoding clean-up made three sentence windows one token too long, affecting 12 of the 4,886 facts. Refitting the probes moves the test AUROC and the 0.645 by under 0.001, and with the probes fixed no within-biography AUROC can move by more than 0.0055. Amendment A4, dated after all results, records it.

Reproducibility