A search for internal signals in Gemma, built and tested by agents

[ 2026-10-23 ] // Claude Opus 5.5 · Gemma 3 1B · Gemma Scope 2 // 15 MIN READ

Agents built a search for the variables Gemma 3 1B tracks as it reads, with tests meant to separate real signals from search artefacts. It recovered parenthesis and quotation state, and one direction passed a preregistered steering test on fresh text. Gemma Scope 2, Google DeepMind's public sparse autoencoders for Gemma 3, already represents that direction, so this is a methods result, not a new variable.

The question, the setting, and who did what

A model keeps track of things as it reads: whether it is inside brackets, whether a clause has finished, whether a negation still applies. The usual ways to find such variables start from something known: labels used to train a probe (a small readout of the model's internal activity), or a trained dictionary's features. I wanted a search that is not told what to look for, with tests strict enough to tell a real variable from an artefact. Once it recovers known variables and passes a causal test, such a search could point at variables neither labels nor a dictionary suggest. It has seven steps:

1. Rules first. Every preregistration, rubric and script is hashed into a ledger before the step it governs. Confirmations use text the search has not read, no verdict is edited later, and a redesign after a failure is labelled post hoc.
2. Remove the obvious, then unmix. Strip out what the token, the document and the position explain, then unmix the rest into independent directions with full-rank ICA, without labels.
3. Check it on known variables. Labels score the frozen directions against parenthesis and quotation state.
4. Name what can be named. A feature battery, a sealed reading of contexts, then mechanical grammar rules.
5. Test causally. Push each direction and compare with same-subspace random directions and other directions' signatures.
6. Confirm on fresh text, with one primary test fixed in advance.
7. Check against the dictionary. Does an existing SAE latent already cover what was confirmed?

The model is gemma-3-1b-pt. The text is FineWeb's sample-10BT stream in 256-token sequences, split into blocks of 1,000. The search runs on the residual stream (the running internal state each layer updates) as reconstructed by Gemma Scope 2's SAE at layer 7 or 12, so every direction it finds is a readout of that SAE's codes.

Claude Opus 5.5 agents in Claude Code designed each step, wrote and debugged the scripts, froze the preregistrations, ran every job on my laptop and kept the record. I set the questions, approved the confirmation runs and decided when to close a line of work, including whether to call anything a discovery. Their work was checked by the hashed rules, the controls, synthetic smoke tests and, before several later freezes, read-only reviews by separate agent sessions; the smoke tests and reviews caught six bugs, two of which leaned toward a GO verdict. No one outside the project has rerun the study.

Finding known variables, and where labels came in

The first instrument failed its positive control: its ICA searched a subspace chosen by noise, and its pass mark (0.85) sat above what a supervised probe could reach (0.83). Version 2 removes token-type, document and position means, whitens with PCA to 64 dimensions, equalises each frequent token type's covariance, re-whitens, and runs full-rank ICA with logcosh and skewness contrasts.

Labels never touched the extraction, but other stages used them. In calibration on a separate block (corpus 13,000–13,999), the agents chose the whitening and screening settings with parenthesis labels and a supervised parenthesis probe, and a rule tied to that probe fixed the pass floor at 0.70. On the discovery block (corpus 1,000–1,999), the directions were frozen before any label was computed, then scored (Figure 1) and named. Quotation state was never used in any choice, so it is the untuned positive control.

At layers 7 and 12, one direction carries 90–97% of a held-out supervised probe's excess over chance. It is a readout, not a clean detector: among a paren unit's strongest 1% of positions, 50–69% are inside parentheses. Token identity is reduced, not gone; nearest-neighbour identity is still 9–15 times chance.

Naming what it finds: often grammar

The discovery run screened 143 directions. A battery of 24 surface features (punctuation, brackets and the like) left 91 unnamed; an agent's reading of their strongest contexts, sealed before any later step, named 26 more, leaving 65 candidates. That run claimed no discovery, because the first steering tier failed its anchors (next section), but 23 candidate families reappeared on a fresh replication block.

The readings kept finding grammatical position, so after the verdict the agents turned each of the 14 into a mechanical rule over a spaCy parse or the previous token. In the real contexts below, the underlined token is where the direction is strongest; the last column is the share of its strongest 1% of positions that the rule marks, against the base rate.

Eleven of the 14 readings held up; the reading also missed four directions its own rules name, so about 19 of the 23 families remain unnamed.

Testing causally, then confirming on fresh text

The first steering tier pushed every direction by a fixed 2% of the residual norm and required it to beat its siblings' largest effect. The siblings carried real effects (the subject-noun-phrase family reached Δ 1.52 at that push), so even the quotation anchor, a direction known to track quotation, failed (0.246). The agents then redesigned the rig and labelled the redesign post hoc:

• the push is scaled to each direction's natural spread, c·σ, with the test at c = 2;
• the null is 99 random directions drawn in the units' own 64-dimensional space;
• a shuffled-signature control scores the same push against 76 other units' signatures.

A signature is the 20 next tokens most over-represented after a direction's strongest 1% of positions, found without labels. Δ is the change in their log-probability against frequency-matched random tokens, steered minus unsteered, averaged over 200 sites, in nats. On two further blocks the quotation anchor passed (Δ 0.856 and 0.873), so the rig works at layer 12; the layer-7 anchor failed, so layer 7 has no behavioural test. One candidate family, logcosh #13 and its twin skew #16, passed on both blocks; those two passes selected it.

A confirmation preregistration, frozen with its script before its text was processed, then fixed one primary test. Push logcosh #13 by 2σ at 200 sites from 1,000 sequences no step of the search had read: does its own signature, unchanged since the discovery block, rise more than under every random direction and more than every other signature does under the same push? The anchor had to pass first.

It passed both controls by a clear margin: its effect is about twice the largest of the 99 random directions (Δ 1.49 against 0.75; their median was about −0.04) and above the largest of the 76 other signatures (0.85). Its mean effect was positive for 19 or 20 of the 20 commonest site token types at every push, and it grows with dose: Δ 0.28, 0.63, 1.49 and 2.88 at 0.5, 1, 2 and 4σ.

The preregistration states a false-pass bound of 1/77. That is a rank bound: it holds if, absent a specific effect, the direction's push moves its signature like any of the 99 random pushes, or its signature responds like any of the 76 others. The design assigned neither the direction nor its signature at random. The family was also selected by its two earlier passes, and under the preregistration's own stable-rank null a unit that has passed twice is more likely to pass again. So I report the margins over the controls, not a probability.

What it does (exploratory)

The confirmation tested only the direction's effect on its fixed signature. Its grammatical meaning was described afterwards and has not been tested as a prediction. Pushed by 2σ, the direction raises words that open an adjunct, a phrase or clause that attaches extra information to a clause. Unpushed, the tokens where it is strongest complete a clause's core (absorb the material in manageable segments), and 28% of the tokens after them are prepositions, against 8% at random.

One real passage, continued unsteered and at three push strengths:

Pushed, the model keeps the clause going; pulled hard, it ends the sentence. The untested name I would give the direction is the clause's core is complete and an adjunct can attach here.

The main lesson: compare with random directions from the same subspace

A screen on a derived direction means little until it is rerun on random directions built the same way, in the same subspace. Five screens in this project proved miscalibrated or uninformative against that baseline: one was too strict, and four set bars that random directions also cleared. In the table, decoder |cos| is the cosine with an SAE latent's output vector, the isotropic null uses random directions in the full space, and p99.9 is the 99.9th percentile.

The confirmation passed against exactly this kind of baseline.

Step 7: Gemma Scope 2 already had it, and why that still validates the method

Step 7's rubric, hashed before any comparison, calls a direction COVERED if one existing latent has decoder |cos| of at least 0.7 with it and passes a behavioural match: at least 30% of the latent's 200 strongest rows are followed by a preposition, subordinator or temporal word. Of 16 Gemma Scope 2 dictionaries checked, three latents qualified, among them layer-12 #11562 (|cos| 0.735, share 0.485).

So it is covered, with qualifications: the cosines sit inside the same-subspace null (Table 03), no single latent tracks it closely (best activation correlation 0.40), and related latents split the behaviour by adjunct type. It is a blend of subordinator and preposition latents that the SAEs keep apart, unified in one direction, not a new concept.

That is still useful validation of the method. Because the search runs on the SAE's own reconstruction, any direction it finds is a mix of SAE latents, so "already covered" was always likely. What was not guaranteed: that a search told nothing about clauses or prepositions would single out one mix with a coherent effect, that the mix would beat same-subspace controls in a preregistered causal test on fresh text, and that the words it promotes would match what follows the strongest positions of existing latents, including latents at other widths and another layer. A method should get known answers right before it is trusted on unknown ones; this one recovered parenthesis and quotation state, passed its quotation anchor, and confirmed a direction the SAEs already encode.

Where the process stops

The process can say a direction is covered. It cannot yet say one is not.

No test certified "uncovered." A single-latent census called 3 of the 4 layer-12 paren and quote controls NOT COVERED, though they have dedicated latents. A knockout control, which removed the SAE's own paren and quote latents, admitted no test at either layer-12 16k dictionary (larger or smaller L0): a probe still recovered paren from the reconstruction after the largest knockout (AUROC 0.663, against a chance band whose 99th percentile is about 0.52), and the random baseline's largest error share climbed from 0.36 to about 1.0. I accepted the closure for layer 12.

Looking outside the span found nothing that survived a matched null. The error-space pilot said GO by its frozen rule, but random directions passed too (Table 03). On the raw residual, the instrument passed its positive control and gave one thin lead, about 76% of which is still SAE span; refit on five shuffles of the same text, the rule that picked it fired once on 4 of the 5.

A behavioural split test was inconclusive. On a fresh block the confirmed direction passed a fourth time (Δ 1.78 against a random max of 1.14) and moved both word classes. Whether its best decoder row moves only one class could not be decided: a precondition compared a bank median with the maximum of 99 heavy-tailed random draws (1.29 against 2.21). I closed that line at 1B.

Limitations

• The process cannot yet certify a new variable. Every direction it found is a readout of SAE codes, the raw-residual run mostly re-found the same ones, and no coverage test passed the knockout control. The next experiment targets this.
• One model. The whole process works only at layer 12: the instrument passes at layers 7 and 12 and fails at 24, and the layer-7 steering anchor failed twice.
• The confirmation ran on a steering rig redesigned post hoc, for a family selected by two passes on it.
• The context readings come from an agent; the rules that check them see whole sequences and depend on a parser.
• Quotation state is the only untuned positive control.

Untested

• A "not a blend" criterion. It needs a positive control known not to be a blend.
• gemma-3-4b-pt, estimated at 9–13 hours on this laptop and deferred until that criterion exists.
• Other coverage bars, knockout ladders at layers 11 and 13, and other dictionaries such as transcoders.

Next experiment

Nothing is scheduled. If the work resumes, the order is:

1. A coverage test that passes the knockout control on already-used text, so that "not covered" means something.
2. Only then 4B, with a disk plan, two protected text blocks and the controls re-admitted at each layer.
3. Before trusting any new instrument, a blind round: one session plants a direction, and another searches for it without knowing where it is.

Related work

• Gemma Scope 2. McDougall et al. (2025) released the open SAEs for Gemma 3 used here.
• Feature absorption. Chanin et al. (NeurIPS 2025) show that latents that look monosemantic can fail to fire where they should when features form a hierarchy. The split seen here is related, but I did not test it as absorption.
• SAE dark matter. Engels, Riggs and Tegmark (TMLR 2025) find that much of the SAE error is linearly predictable. The error pilot here found nothing context-specific beyond what random directions show.
• Stitching and meta-SAEs. Leask et al. (ICLR 2025) and Bussmann et al. (2024) show that SAE latents are neither canonical nor atomic. That fits single-latent coverage failing here.
• Reconstructions and steering. Kharlapenko et al. (ICML 2025) report that passing a task vector through an SAE noticeably reduced its effect. The split test here asked a related question.

Reproducibility