Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects

SAE features that activate on a single vocabulary token are the closest thing mech-interp has to a ground-truth-verifiable benchmark. This paper runs causal ablation tests across 3.9 million features from six models and three SAE families, finding that which family you train on matters more than model scale — and that LlamaScope features are locally redundant where GemmaScope and BatchTopK features are causally anchored.

The authors use single-token SAE features — those activating on exactly one vocabulary token — as ground-truth-verifiable causality diagnostics: if ablating the feature doesn't reduce log-probability for that token, the feature is not genuinely causal. Testing 3.9M features from six models across GemmaScope, BatchTopK, and LlamaScope SAE families, single-token features cluster 4.7× tighter in decoder space and concentrate in early layers; ablation yields BH-significant logit reductions in 178 of 208 conditions. The critical result is that the cross-family effect size exceeds the within-family scale effect — GemmaScope and BatchTopK features are causally anchored while LlamaScope features are locally redundant. The implication for interventional mech-interp: SAE family selection is a more important methodological choice than model scale.