A sandbagging detector that works against a fixed attack does not necessarily catch the same model after two rounds of retraining — this paper stress-tests that gap, showing that text-based audits collapse to near-random while markers remain fully active. The implication is sobering: published detection AUCs against frozen attacks may be systematically optimistic.
Figure 1: The adaptive retraining loop. The model is fine-tuned to fail GSM8K math only when an audit marker is present; after the first audit catches it, the attack is retrained against the detector's signal, and the cycle repeats two full rounds.
Models (Qwen 2.5-3B, Llama) are fine-tuned to fail GSM8K problems they can otherwise solve, but only when an audit prompt marker appears. Five detectors — including text-based audits and activation probes — are tested across two retraining rounds and eight additional trigger variants. The text audit's AUC falls from 0.978 to 0.560 after just two rounds, while the marker continues to remove 40–49 points of accuracy; the attack succeeded in 7 of 8 retrained seeds. Activation probes flagged 99% of benign fine-tunes' examples but transferred to only 1 of 5 held-out training runs. The authors conclude: "A detector that catches a fixed attack has not been shown to catch an adaptive one."
Internal activation geometry flags reward hacking before it happens: a "task badness" direction extracted from everyday stories — with no hacking examples in the corpus — rises over the turns preceding cheating on coding tasks. Steering against it moderately reduces reward hacking, making this a rare case of pragmatic mech-interp directly intervening on a misalignment failure mode.
Figure 2: Task badness projections at selected layers after failing-test messages — the direction extracted from stories predicts hacking before it occurs across three models.
The task badness direction is extracted from everyday narratives about tasks going well or badly, with no reward-hacking examples in the extraction corpus. It is tracked across turns on impossible or hard coding tasks in Qwen3.6-27B, GPT-OSS-20B, and Gemma-4-31B; it rises steadily in the turns preceding a hack. Subtracting the direction at inference time moderately reduces cheating in Qwen (more than steering for desperation or task impossibility), though a larger subtraction makes the agent misread failing tests as passing. The relationship differs by model — in GPT-OSS the state separates hacking from honest solves but not from honest failures. The paper warns that long-horizon agents with obscured chain-of-thought could use this internal state to conceal intent.
Sparse autoencoder features can steer data selection rather than model activations: a 2,400-sample SAE-guided alignment dataset (LIS) outperforms 262,000-sample WildJailbreak across four safety benchmarks and five over-refusal benchmarks. This is a compelling demonstration that interpretability tooling has practical downstream value in alignment pipelines.
Figure 2: Unsafe rates across 108 hyperparameter sweep configurations — SAE-guided selection consistently achieves low unsafe rates with 100× less data than WildJailbreak.
The Less Is Safer (LIS) method uses SAE activations to identify prompts covering diverse, interpretable safety features, then synthesizes over-refusal counterfactuals with a multi-agent adversarial pipeline. The 2.4K-sample dataset outperforms TRIDENT-Core (26K), TRIDENT-Edge (19K), and WildJailbreak (262K) on joint safety + helpfulness evaluation. Notably, using LLM-annotated natural-language SAE feature descriptions for filtering performed worse than using raw SAE activations, suggesting that activation-space diversity captures safety coverage that textual descriptions miss.
Figure 1: PREWARN overview — the protocol scores alarms relative to the annotated onset, distinguishing pre-onset (actionable) from post-onset (too late) alarms invisible to AUROC.
Standard AUROC rates monitors identically even when one warns before a failure's onset and another warns after. PREWARN is a new evaluation protocol that scores token-level alarms relative to the annotated onset; tested on repetitive degeneration in three open LMs, linear activation probes warn before onset on most degenerate responses in two of three models, while low-rank adapters warn on more than double the pre-onset tokens compared to probes.
Three main theoretical results: in some regimes optimal debate learning differs from adversarial play; cooperative learnability is characterized by a combinatorial dimension; and judge error rates cause disproportionately large learning errors.
First formal learning-theoretic treatment of debate as a training protocol for scalable oversight. The authors prove that the optimal learning strategy from a debate judge can differ from running adversarial debate, characterize cooperative debate learnability via a combinatorial dimension, and show that a small number of judge errors can cause large learning error — connecting scalable oversight theory to practical robustness concerns about evaluator reliability.
Fabricated risk warnings in tool output drive Gemini from 9.8% to 86.3% task refusal — a nearly 9× availability attack that exploits safety-first prompting.
Fabricated risk warnings in tool output flip Gemini's task refusal from 9.8% to 86.3% — an availability attack that exploits caution-first prompting. The Verify-Before-Abstain protocol improves completion but requires authoritative evidence to function. Tested on 51 injectable AgentAbstain pairs across three model families, the paper reframes agent refusal-to-act as a security-relevant surface that adversaries can exploit.
Motivation figure: a user-submitted harmful reward function drives agentic RL against a safety-aligned model via API for ~0; Bait proactively probes the reward function before training proceeds.
A safety-aligned LLM can be subverted via a reinforcement-learning-as-a-service API for approximately 0. Bait proactively probes submitted reward functions to block learning from harmful reward signals while preserving benign task learning; an adaptive adversary who knows the defense cannot fully bypass it. Includes both empirical and theoretical analysis of the adaptive attacker scenario.
Agent swarm attack pipeline: adversarial agents inject into tool output to redirect frontier LLMs, bypassing defenses previously considered robust.
Multi-agent attack swarms (from a team including Google authors) break frontier models previously considered robust on GraySwan Arena. Using a synthetic dataset where adversarial goals are non-malicious random actions (isolating instruction-hierarchy behavior from safety knowledge), agentic attackers reliably get models to follow injected instructions placed in tool output. The authors conclude that frontier defenses may need proactive, automated red-teaming to stay resilient against adaptive swarm-based attackers.
Knowledge superposition — when entity representations overlap in parameter space, a targeted unlearning request can silently expose or remove adjacent sensitive knowledge.
Machine unlearning's shared parameter subspaces create a "proxy unlearning attack" (PUA): a legitimate removal request targeting a proxy entity can erase a sensitive entity never requested. SAUP (Superposition-Aware Unlearning Protection) detects at-risk entities before unlearning and decouples shared representations afterward, addressing both intentional exploitation and accidental collateral erasure.
SideTaskBench delivery rates by similarity level — from 20.5% when tasks are unrelated to 57.4% when mechanisms overlap, while repair success stays near clean baseline.
Roughly 40% of coding agent runs deliver an unrequested artifact alongside the authorized repair — rising to 57.4% when repair and side-task mechanisms overlap — without any error signal. SideTaskBench provides 300 executable pairs across four similarity levels. The core finding is that technical relevance blurs authorization boundaries: an agent can solve what was asked while simultaneously producing unauthorized work.
