SAE interpretability on GPT-2: first results
What I found after running feature exploration and causal interventions on a pretrained sparse autoencoder. One feature surprised me.
TL;DR: One feature in GPT-2-small's pretrained SAE — #9577 — steers generation toward a cluster of marginalised groups: patients, prisoners, juveniles, addicts, and also homosexuals, Iraqis, Somalis. It fires on clinical prepositions in medical text, not on those group nouns themselves. Working hypothesis: GPT-2 has learned a single representation for "people described as objects of institutional attention" — a concept that spans medicine, incarceration, and social marginalisation because they share a register in the training data. The causal signal is real at low steering strength (α=5). The interpretation is a hypothesis, not a confirmed result; the fire-vs-steer mismatch is the interesting open question.
Phases 1 and 2 of the SAE interpretability project are done. Here's what actually happened — including what didn't work and one finding that genuinely surprised me.
Setup
The stack: GPT-2-small (124M params, 12 layers, d_model=768), a pretrained SAE from Joseph Bloom's work (gpt2-small-res-jb), hooked at layer 8's residual stream (blocks.8.hook_resid_pre). The SAE has 24,576 features — a 32× expansion of the 768-dim residual stream.
Phase 1 explored 20 of those 24,576 features. Phase 2 tested whether the labels I gave them were actually causal.
The 20 weren't hand-picked for interest. The script sampled 400 features at even spacing across the full index (every ~61st feature), filtered to those with density between 0.2% and 15% — the range where individual features tend to be specific enough to interpret — sorted by density, and took the top 20. Feature selection happened before looking at any examples.
How feature exploration works (Phase 1)
For each feature, two independent signals:
Logit-lens — purely mathematical. Take the feature's decoder direction (a 768-dim vector), project it through GPT-2's unembedding matrix. The top tokens in that projection are what this direction "votes for" at the output. Fast, but unreliable — it ignores LayerNorm and that we're at layer 8, not the final layer. A tuned lens (a small probe trained per layer to project more accurately to vocab space) or applying the final LayerNorm before projecting would both reduce this error; this is a known first-pass approximation.
Max-activating examples — empirical. Run 2,000 real text documents through GPT-2, record the SAE feature's activation at every token position, find the 10 tokens where it fired hardest, show surrounding context. Slower, but honest. When the two methods disagree, trust this one.
What I labelled and what held up
Five features I was confident enough to label:
| Feature | Density | Label |
|---|---|---|
| #7137 | 3.24% | Code — XML tags, config file syntax, dot notation |
| #488 | 1.07% | Code/markdown structure — imports, ### headers, indentation |
| #16836 | 1.12% | Pivot/consequence words ("so", "therefore", "thus", "instead") |
| #13481 | 1.47% | Mislabelled initially — see below |
| #9577 | 1.37% | Marginalised people as objects of institutional description (hypothesis — see below) |
The density column matters: as a rough heuristic, features firing on 1–4% of tokens are in a useful range — specific enough to have a distinct concept, common enough to find examples. Above 10% and the feature fires on so much it's probably too vague to label. These thresholds come from practice, not theory.
What Phase 2 actually tests
Clamping: add alpha × decoder_direction to the residual stream at every token during generation. Forces the feature on artificially. If my label is correct, the model should start generating text that matches the concept.
Ablation: remove the feature's contribution from the residual stream at every token. If the feature was naturally firing on my prompt, removing it should shift the output.
The causal claim is: if clamping steers toward the label AND ablating shifts away from it, the feature isn't just correlated with the concept — it's causally upstream of it.
The clamping effect is measured as a boosted-token table: for each candidate next token, compute the raw logit under clamping minus the raw logit at baseline, at the last prompt position. A score of +7.31 for "juveniles" means the model's log-probability for generating "juveniles" next increased by 7.31 when the feature was clamped on at α=40.
The mislabelled feature (#13481)
I labelled #13481 as "modal necessity/obligation" based on one example: "it may be necessary to perform a joint replacement." Seemed reasonable.
Then I clamped it. The boosted tokens were: usually, invariably, depending, sometimes, beforehand, whichever.
Those aren't obligation words. They're frequency and hedge words. The feature is about how often things happen, not whether they're required. One example was enough to build a wrong hypothesis. The logit-lens hint had said "usually, often, invariably" which I'd ignored in favour of the example. It was right.
Corrected label: frequency/habitual language.
The surprising finding (#9577)
Feature #9577 had logit-lens tokens: inmates, patients, prisoners, detainees, captives. I labelled it "medical/clinical text" because the max-activating examples were all hospital contexts — "Apollo Hospital in Delhi", "34 patients undergoing", "ESRD in 50% of patients by 60y."
When I ran the causal test with prompt "After surgery, patients with severe" at α=40, the top-10 boosted tokens (next-token logit difference, clamped − baseline) were:
+7.31 juveniles
+7.19 prisoners
+7.16 inmates
+6.31 detainees
+6.08 Iraqis
+5.96 youngsters
+5.83 offenders
+5.81 addicts
+5.71 homosexuals
+5.67 migrants
The fire-vs-steer gap
In a well-behaved monosemantic feature, what fires it and what steering reveals should be the same concept. Here they aren't:
| What fires this feature | What steering reveals |
|---|---|
| Prepositions ("of"/"in") in clinical/research text | Prisoners, detainees, juveniles, addicts |
| 68% medical, 20% animal research contexts | Homosexuals, Iraqis, Somalis, migrants |
| 0% prison, 0% detention, 0% migration | — absent from firing examples entirely |
That gap is the central puzzle of this finding. It would be easy to paper over it by just picking one side ("it's a medical feature", or "it's a confinement feature") — but neither fits the full evidence.
Not just medical text — and the picture is more complicated than my first interpretation.
Running 50 max-activating examples, the feature fires 68% on medical/clinical research text and 20% on animal research — and the token that peaks is nearly always a preposition: "records【of】patients", "brachytherapy【in】comparison", "kidneys【of】mice subjected to UUO". Not the vulnerable-group noun itself. The feature appears to encode a syntactic expectation: an institutional subject follows here.
But the steering signal reveals something much broader. The boosted tokens include not just institutionally confined people (prisoners, detainees, juveniles, addicts) but also groups that are marginalised without being confined: homosexuals, Iraqis, Somalis, migrants. Zero of those groups appear in the max-activating firing examples.
The best current hypothesis: GPT-2 has internalised a representation where patients, prisoners, lab animals, and minority groups share a common register — all described as objects of institutional attention in the training data. If correct, this would be a bias/representation finding: the model has collapsed disparate social categories into a single learned direction because they tend to appear in the same depersonalised, institutional language in training text.
It's a hypothesis, not a confirmed result. What the evidence does establish: the feature is causally upstream of something real (clamping and ablation both shift output consistently), the steering effect appears at α=5 (in-distribution, not an artefact of extreme intervention), and the same cluster appears across six diverse prompts. What it doesn't establish: whether "objects of institutional attention" is the right frame or a plausible story fitted over a simpler pattern (syntactic register, vocabulary overlap in the training data, rare co-occurrence); and whether this generalises beyond this one SAE trained at this one layer.
The generation diffs
Best before/after (prompt: "Apollo Hospital in Delhi have saved the life of", alpha=40):
baseline: "...a young girl who was shot dead by a man who was trying to rob her."
clamped: "...a man who was injured in the same. The man was taken to the hospital."
ablated: "...a young woman who was shot in the head by a man trying to rob her."
Three distinct outputs. Baseline generated a crime narrative about a girl. Clamped shifted into a hospital narrative. Ablated changed "girl" to "woman" — the feature was contributing something specifically about youth/vulnerability, and removing it nudged toward a more generic adult.
A real causal result — but with an asymmetry worth naming. Clamping produced a large, consistent effect across six diverse prompts at multiple alpha values. Ablation produced a subtle one-word shift (girl → woman) on the single prompt where the feature fired hardest naturally. The direction fits the label; the magnitude is mild.
That asymmetry is expected: clamping forces the feature far above its natural activation level, so the effect is large. Ablation removes a contribution that was already present at its natural level — which may be modest. But it also means the ablation side of the causal claim is weaker than the clamping side, and should be read as supporting evidence rather than independent proof.
The negative direction (α=-40)
If positive clamping (α=+40) steers toward vulnerable/institutional people, negative clamping (α=-40) subtracts that direction at the same magnitude. The question is: what fills the space when the feature is actively suppressed?
Two generation diffs stand out:
prompt: "At the border, the officials saw"
baseline: "...a man with a gun and a knife."
α=-40: "...a bright light of the day, and they were ready to go."
prompt: "After surgery, patients with severe"
baseline: "...pain and swelling...walk...doctor..."
α=-40: "...pain and swelling of the lower back...acetaminophen-containing acetaminophen-containing..."
In both cases, the human subject disappears. The border scene loses the armed man entirely; the surgery output stays medical but shifts toward pharmaceutical product language, with the patient becoming less present. The boosted-token table (next-token logit increases under suppression) is dominated by code identifiers and commercial terms: dayName, inventoryQuantity, EntityItem, dealership, Insurance, warranty.
The negative pole of the feature looks like non-human, technical, transactional content — code database fields, commercial products, no persons. Consistent with the hypothesis that the feature encodes a dimension from "impersonal/technical" to "human subjects of institutional attention."
One caveat: SAE features use ReLU activations and are only trained on non-negative values. Forcing the feature to -40 pushes the residual stream into territory the SAE has never represented. The boosted-token table reflects this — it includes garbage tokens (control characters, byte fragments, partial corporate names like isoft for Microsoft) alongside the meaningful commercial terms. The generation diffs, which go through GPT-2's full forward pass, are more robust. Read the generations as evidence; treat the token table as directional but noisy.
What didn't work
Feature #7137 (code/XML syntax) showed almost no steering effect even at alpha=20. The boosted tokens were proper nouns (Myst, Leone, Blaz) — not code tokens. The logit-lens hint was actively misleading.
My interpretation: #7137 is a document-type feature. It fires when the model is already inside a code document, signalling context. That's different from being causally upstream of code token generation. Forcing it on a prompt like "I am going to gym" doesn't produce code — there's no scaffolding for the model to build on.
One bug found
The pretrained SAE (SAELens) stores its decoder weight matrix as (d_sae, d_model) = (24576, 768) — each row is a feature direction. The codebase was written assuming (d_model, d_sae) — each column is a feature direction. The index W_dec[:, 7137] was trying to select column 7137 from a 768-column matrix and crashing.
Fix: detect the layout from the shape (shape[0] > shape[1] means SAELens layout, use row indexing) and apply the right one. Two files needed patching: features.py and interventions.py.
Next
Phase 3 is training my own SAE from scratch on the same activations — the SparseAutoencoder class I wrote, not Bloom's pretrained weights. The goal is to confirm I understand the architecture well enough to reproduce the phenomenon, not just analyse someone else's result. Needs a GPU; running on Colab.