Hidden activations know when self-report does not

When self-report is not enough

Long answers fail locally. One sentence is right, the next is partly right, the next is confidently wrong. A user who depends on the answer needs to know which is which. A single answer-level confidence number is not enough.

The simplest claim-level baseline is to ask the model. Show Llama 3.1 8B a fixed claim from its own answer and ask for structured confidence. If self-report ranks correct claims above incorrect ones, a probe-based system is wasted complexity. If it does not, the alternative has to come from somewhere else inside the same forward pass.

The probe is deliberately simple: layer-19 residual-stream vectors pooled over the claim span, standardized, fed to L2 logistic regression. No attention head selection, no SAE feature search, no learned pooling. The point is to test whether the basic geometric assumption holds before reaching for anything more expressive.

Setup — one probed model, three label regimes

The probed model is always Llama 3.1 8B Instruct. What changes is the dataset and label source. MAUD provides 150 fixed legal claims scored by two LLM judges. FActScore provides 4,886 human-labeled atomic facts mapped to parent biography sentences. FELM provides human segment labels on world-knowledge QA, where the matched-generation repair eventually exposed an annotation-target problem.

The two protocols differ because the sample sizes do. MAUD's 150 claims do not support a clean train/validation/test split, so the headline MAUD probe uses leave-one-out evaluation over the frozen full set. FActScore's 4,886 facts comfortably support a 70/15/15 split by biography, with 3,353 train, 791 validation, and 742 test claims. Both choices are conventional for their respective regimes; the comparison the article makes is over the qualitative gap, not over identical pipelines.

The headline — same gap, two domains

The simplest evaluation is whether each score ranks correct claims above non-correct claims. AUROC is 0.5 at chance. Higher is better. Brier score on the same scores measures how well the probability magnitudes match the binary label.

Why this is a replication, not a copy

MAUD uses leave-one-out over a small judge-labeled legal claim set. FActScore uses a 70/15/15 split by biography over a much larger human-labeled biography set. The protocols are not identical, and they should not be: 150 rows do not support what 4,886 do. What replicates is the qualitative gap between self-report and residual activations, with paired effect sizes of +0.260 and +0.262 AUROC. Treat the agreement as evidence about the signal, not the pipeline.

Second judge — does the ordering survive?

MAUD's first labels come from a GPT-5.4 judge-proxy protocol. Probes trained on those labels could be fitting whatever GPT-5.4 happens to score consistently rather than something more general. The cleanest test is to relabel the same fixed claims with a different judge family and rescore the same probes without retraining. We use Kimi K2.6 through Prime Intellect Inference, with the prompt structure and claim serialization preserved.

Where the judges actually disagree

The two judges agree on 110 of 150 claims. Cohen's kappa is 0.572 [0.456, 0.682]. The disagreement is not uniform across the rubric. Stratifying by GPT-5.4's label shows where the boundary is hardest.

The judges agree most on clearly true claims, less on clearly false claims, and least on partially-true claims. That is exactly where the rubric boundary should be hardest, and it is the regime that drives the kappa down toward 0.57. The 110-claim agreement subset is therefore not just a robustness slice; it is the most interpretable subset MAUD has, because it strips out labels where two different judges meaningfully disagree.

FActScore probe diagnostics — not a fragile fit

FActScore changes everything except the probed model: human atomic-fact labels instead of LLM judges, ChatGPT generations instead of Llama generations, biography factuality instead of merger-agreement QA, atomic facts pooled to parent sentence spans instead of extracted legal claims. Despite all of that, the residual probe's held-out AUROC lands at 0.802. The diagnostic question is whether that test number reflects a real generalization or a happy validation set.

The Brier scores tell the same story without the rank-only filter. Self-report Brier is 0.298, which means the probabilities Llama emits are barely better than always saying 0.5; the residual probe Brier is 0.171, reaching well past calibration parity. AUROC and Brier disagreeing would be a flag for rank-vs-magnitude trade-off; here they agree, and the probe wins on both.

The boundary — when labels measure coverage

FELM-wk is the third dataset, and it is the cleanest case for the cautionary half of the article. The minimum-viable run reaches 0.652 residual probe AUROC against 0.511 for self-report, with paired delta +0.141 [-0.063, 0.320]. Direction is positive. The interval crosses zero. That is what "directionally consistent but inconclusive" looks like in numbers.

The interesting finding in FELM is not the held-out AUROC. It is what happened when we tried to repair the annotation. Llama-generated FELM-style answers go beyond FELM's curated reference snippets often enough that calling them false against those snippets is the wrong target. We added fuller readability-extracted references and a fourth not_enough_evidence label. The repaired pilot's 52 segments came back like this:

not_enough_evidence at 57.7% is not noise. It is a structural finding: when generated claims exceed the reference scope, a reference-bounded annotator cannot adjudicate them. The label tells us whether the bundle supports the claim, not whether the claim is true. That is a coverage measurement, not a correctness measurement, and it is the wrong target for a probe trained on claim correctness. We therefore did not scale the matched-generation FELM run; instead, FELM stays in the paper as a documented boundary that explains why label provenance matters more than domain similarity.

The methodological lesson

The strongest lesson is not "legal QA transfers to biography QA." It is that label provenance dominates. MAUD works because fixed claims are judged by a clear rubric. FActScore works because human atomic-fact labels directly capture factual support. FELM is weaker because segment labels and reference coverage are coarser than the claim-level correctness target the probe needs. Claim construction and label provenance affect probe trainability as much as the probe architecture does.

For a probe-based uncertainty system, the hard questions come before the classifier ever runs. What is the claim? What does the label actually capture? Does the activation span match the label unit? When those three line up, a regularized linear readout of the residual stream recovers a sizeable correctness signal. When they do not, a stronger probe will mostly learn a muddier target.

Bounds on the claim

What's next

Three follow-ups would most sharpen the claim, in roughly this order.

Close the human-audit loop

The 30-claim MAUD audit packet is frozen at data/annotations/maud_human_audit_packet.jsonl. Completed expert labels would convert MAUD's claim from "stable across two LLM judges" to "stable across two LLM judges, anchored to expert legal labels," or expose a systematic disagreement worth knowing about. Either outcome is publishable.

Matched-generation transfer that does not collapse

FActScore weakens the generation-mismatch worry but does not eliminate it. A matched-generation run on a dataset where Llama's claims do not routinely outrun the reference scope would close that gap. The FELM repair shows the failure mode to avoid: reference-bounded labels driven into not_enough_evidence.

Layer sweep and second model family

All probes are layer 19, chosen because the Goodfire SAE was trained there. A small layer sweep would document how localized the signal is. A second model family at 7-13B parameters would say whether the gap is a Llama representation quirk or a more general property of instruction-tuned residual streams.

The result is a bounded interpretability win. A linear readout of hidden activations sees something the model's own structured confidence misses, and the gap is the same size on legal QA and on biography factuality. It does not give us a compact SAE feature story, a legal-truth adjudicator, or a domain-general hallucination detector.

The remaining questions sit on label provenance and matched-generation transfer, not on probe architecture. Downstream tools building on this should expose claim construction and label-provenance assumptions to users, rather than hiding them behind opaque confidence scores. The probe is only as honest as the labels it was trained on.