Cross-paper SAE comparisons hinge on autointerpretability scores — but this ETH Zürich paper runs the audit those comparisons have needed: methodological variance from pipeline choices (corpus sampling, evaluator model, prompt template) collectively exceeds the variance explained by SAE architecture across all four tested metrics. Rankings change when the pipeline changes; the architecture signal is buried.
Figure 3: Proportion of total score variance attributable to pipeline configuration choices vs SAE architecture, across four autointerpretability metrics. Pipeline variance dominates all four metrics; fuzzing is dominated almost entirely by pipeline effects (89%), making cross-paper comparisons using this metric uninformative.
Experiments span four autointerpretability metrics (simulation, detection, fuzzing, purity), two models (Pythia-160M and Apertus-8B), and four axes of methodological variation: corpus sampling strategy, draw conditions, evaluator model choice, and prompt template. Three results. R1: methodological variance collectively exceeds architectural variance across all four metrics on both models. R2: each metric has a distinct instability profile — detection is the most stable and suitable for cross-paper comparison; fuzzing is unreliable across all conditions. R3: top-k feature rankings do not stay consistent across corpus and draw conditions, meaning aggregate scores paper over per-feature variance that matters for mechanistic claims. Practical implication: any SAE paper using autointerpretability scores as its primary evaluation needs an ablation over pipeline variables to isolate the architectural signal; papers that do not run this ablation cannot rule out that their observed improvements are pipeline artefacts.
Figure 5: MAP-Elites behavioral grid for GPT-4o-mini showing fitness (attack success rate) across strategy × encoding dimensions. Hypothetical framing + ROT13 encoding achieves fitness 0.8 — the discovered winning combination. Claude shows uniformly low fitness (max 0.4) across all cells, indicating a qualitatively different safety geometry.
Applies MAP-Elites evolutionary search to LLM red-teaming, operating over interpretable semantic attack strategies (behavioral dimension: strategy type × encoding × prompt length) rather than token sequences. An archive grows with diverse attacks; fitness measures jailbreak success rate. ICLR 2026 Agents in the Wild Workshop. Discovers distinct vulnerability profiles per model — GPT-4o-mini vulnerable to hypothetical+ROT13 (fitness 0.8), Gemini to direct+ROT13+Leetspeak (0.8), Claude uniformly low (max 0.4) — surfacing each model's safety geometry structure alongside attack examples.
Figure 6: R2LM (Bifocal dLLM) resolves the KV-cache dilemma with asymmetric design: standard causal attention provides left context with full caching compatibility; a Mamba state-space layer provides position-aware right context without quadratic recomputation. Together they match bidirectional-attention quality while preserving AR-level serving throughput.
Full bidirectional attention in masked dLLMs gives high generation quality but prevents KV caching (each denoising step recomputes attention over the full sequence), degrading serving throughput. R2LM (Bifocal dLLM) uses causal attention for left context — fully KV-cache compatible — and a Mamba state-space model for continuous position-aware right context, without quadratic cost. Matches bidirectional-attention quality on generation benchmarks while achieving serving throughput competitive with AR models. (Meta AI / UNC Chapel Hill)
Figure 8: Standard inner-product SAEs (left) waste dictionary slots on norm detectors — features that activate on token input magnitude, which sublayer normalization has already discarded. Cosine-scored SAEs (right) learn a blend that optimizes toward pure cosine similarity, filling all dictionary slots with content-aligned features.
Standard SAE activation scores depend on both feature direction and input norm, but sublayer normalization discards magnitude before the residual stream — so norm-sensitive scores detect a quantity the model doesn't read. Naihin and Stambler replace the inner-product encoder with a learned blend of cosine similarity and input magnitude; training always recovers near-zero magnitude dependence. At matched reconstruction loss, cosine encoders learn features that align with human-recognizable semantic concepts measurably more often, filling dictionary slots that inner-product encoders waste on norm detectors. The authors argue cosine scoring should be the default for dictionary learning on layer-normalized representations.