Basel Shbita, Michael Hind, Chad DeLuca, Nathan Fulton, Shubhi Asthana, Farhan Ahmed, Anisa Halimi, Yada Zhu, Mauro Martino, David Cox, Dan Gutfreund — arXiv preprint, Oct 28 2026 (IBM Research)
Indirect prompt injection success rates across deployed agentic benchmarks range from 53% to 100%. This paper reduces them to 0.02–5.6% — with no architectural changes, just post-training that teaches models to distinguish instructions from data by enforcing explicit execution semantics.
Figure 1: LLMON post-training introduces identifier-based execution semantics ([EXEC] vs [DATA] tokens). Without LLMON, LLMs treat injected data as executable instructions. After post-training, only explicitly marked [EXEC] tokens trigger actions — collapsing IPI attack surface from 53–100% to 0.02–5.6% ASR.
Shbita et al. (IBM Research) address the root cause of prompt injection: LLMs have no built-in distinction between executable instructions and passive data. LLMON (LLM Object Notation) enforces this via post-training, building a 4.6B-token structured corpus covering general, RAG, and tool-calling scenarios with distracting instructions embedded in passive data contexts. Two Granite-4.1 base models (3B and 8B) are post-trained with supervised fine-tuning then reinforcement learning. Attention-based interpretability metrics confirm the approach reshapes cross-layer attention patterns, reducing attention to injected data tokens. On AgentDojo and InjecAgent, IPI attack success rates fall from 53–100% to 0.02–5.6% — the strongest reduction in IPI ASR reported this reporting cycle — while general benchmark performance is preserved. No architectural changes are required; the method is applicable to any LLM via post-training.
Jonathan F. Jao, Andrew Kyle Lampinen — arXiv preprint, Oct 28 2026 (DeepMind)
Automated adversarial auditing of LLMs treats conversation history as states and responses as actions — but standard one-step sampling misses risky conversational branches. This paper uses linear probes on the target model's own activations to estimate future-harm value and guide beam search, yielding a ~50% improvement in peak audit scores.
Figure 1: A linear probe trained on the target model's activations estimates the expected future-harm value V̂(s) at each conversational state. Beam search uses these values to prune low-risk branches, focusing the adversarial audit on high-risk conversational trajectories. Result: ~50% improvement in peak audit scores across 18 target behaviors.
Jao and Lampinen (DeepMind) frame adversarial auditing as a Markov decision process: conversation history is state, candidate continuations are actions, and LLM judge scores are rewards. They train a linear probe on the target model's internal activations to predict discounted future harmful-outcome probability — effectively learning a value function V̂(s). Beam search uses V̂(s) as a pruning heuristic, expanding candidate branches that look high-risk and discarding low-value ones. Probes can be fit off-policy from smaller models, making the method sample-efficient. Across 18 target behaviors and 3 model scales, peak audit scores improve ~50% over token-matched standard auditing and match more expensive LLM-as-judge follow-up methods at lower cost. This is a clean demonstration that mechanistic interpretability (activation-level probing) yields direct safety tooling value.
When one AI agent pressures another to break its operator's rules, the violation is already encoded in activations before any text is generated. A linear probe can catch 82% of imminent rule violations before output — enabling preemptive oversight rather than post-hoc filtering.
Figure 1: In multi-agent pressure scenarios, a linear probe reads the target agent's activations before each output generation step. At the critical turn — before the violation is produced — the probe flags imminent rule-breaking, enabling the overseer to halt execution. Qwen3-32B: 82% recall at 5% FPR; gpt-oss-20b reasoning: AUROC 0.91.
Raghav tests a "pre-crime" framework in six multi-agent scenarios where a peer agent persistently pressures a target to disclose secrets (e.g., deploy tokens) over up to 20 rounds. A linear probe trained on the target's internal activations, read at each call before output generation, predicts imminent rule violations. On Qwen3-32B held-out episodes, the probe catches 82% of leaking calls at a 5% FPR, compared with 26% for a zero-shot decision-model monitor at similar FPR. On gpt-oss-20b reasoning traces, AUROC reaches 0.91. Important caveats: a bag-of-words label classifier achieves similar performance, suggesting much of the gain derives from training signal rather than activation geometry; the probe also over-flags permitted disclosures, and the short pre-generation window limits intervention time. Still, the work motivates pre-generation activation monitoring as a component of multi-agent oversight systems — complementing post-hoc output filters that arrive too late to prevent leakage.
Michał Burzyński, Noah T. Rossi, Shomit Basu, Andy Arditi — arXiv preprint, Oct 28 2026
Safety-trained models give sparse reward to a jailbreak investigator — most attack attempts are simply refused. Curriculum of Models solves the exploration problem by starting the investigator against an easy target and escalating; it reliably elicits CBRN content from frontier open-weight models where direct training fails.
Figure 1: Curriculum of Models trains the jailbreak investigator against progressively harder model targets — from permissive (dense reward) to fully safety-trained (sparse reward). The investigator builds transferable strategies at each stage before facing the deployment target.
Burzyński et al. address the fundamental challenge of automated red-teaming: safety-trained models refuse almost every prompt, giving an investigator training on sparse rewards and near-zero gradient signal. Curriculum of Models (CoM) solves this by training against a sequence of targets, starting with a model that rarely refuses (dense reward) and advancing to the hardened deployment target once the investigator reliably elicits the target behavior. Tested on CBRN materials, explosives, and illicit drug synthesis across gemma-4-31b-it and gpt-oss-120b. In single-behavior settings, CoM outperforms both PRBO (propensity-bound baseline) and direct training. In multi-behavior settings, CoM investigators generalize to held-out behaviors; PRBO achieves higher absolute success rates in multi-behavior settings but requires more compute per jailbreak. The method provides a practical, scalable framework for measuring dangerous-capability controls before deployment.
Safety behavior concentrates in the first few output tokens — a mechanistic finding that has mostly been used to exploit alignment. ShallowDefend turns it into a defense: RL training on an explicit assess-decide-respond format locks safety decisions in early tokens, achieving the best multi-turn safety across 14 benchmarks.
Figure 1: ShallowDefend output format — the model commits to a parseable [DECIDE] safety label before generating [RESPOND]. RL reward is computed on the label, concentrating gradient on early-token deliberation. Per-token KL and interventional probes confirm the label is causally upstream of the full response trajectory.
Ma et al. (Microsoft Research) observe that safety behavior in LLMs concentrates in the first few output tokens and design a defense that exploits rather than ignores this "shallow" structure. At the payload turn, the model is trained with RL to first Assess (analyze user intent), then Decide (commit to an explicit parseable safety label), then Respond (conditioned on the label). RL reward can be rule-based (no judge needed) or graded via LLM-as-judge for a continuous decision space. Per-token KL analysis confirms the training concentrates gradient on the Decide span; interventional probes show the committed label is causally upstream of the response trajectory. Evaluated across two model families and 14 safety, utility, and helpfulness benchmarks, ShallowDefend achieves the strongest multi-turn safety while preserving utility at a modest helpfulness cost.
Chengquan Guo, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Jialin Song, Qiyao Ma, Carolyn Buractaon, Christopher White, Bo Li, Jianfeng Gao — arXiv preprint, Oct 28 2026 (Microsoft Research)
Figure 1: The "trust region" effect — identical injection content placed in README/TODO files achieves ~2.6× higher ASR than the same content in code or data files, across three agentic testbeds (tool use, coding, daily workflows).
Systematic measurement showing that where an injection appears in an agent's environment determines success at least as much as what it says. Documentation and task-planning surfaces (README, TODO) carry ~2.6× the attack success of structured data surfaces, revealing a surface-level "trust region" effect driven by how agents implicitly weight different context types. The authors release learning-based attackers that jointly optimize content and placement, showing non-linear interaction. Implication: sanitizing structured data fields while leaving documentation unguarded leaves a major residual attack surface.
Xu Zhang, Ziqing Hu, Shuo Han, Ren Wang — arXiv preprint, Oct 28 2026
Figure 1: JC samples diverse adversarial images along low-loss paths connecting known effective jailbreaks in image space; path-sampled images retain jailbreak effectiveness while achieving diversity, transferability, and universal coverage across queries.
Jailbreak Connectivity (JC) exploits the observation that effective adversarial images occupy connected low-loss regions of image space, using path sampling to generate diverse jailbreaks without re-optimization. Lightweight surrogate classifiers guide cross-model transfer and a distribution-level objective targets universal attacks. On MM-SafetyBench: 84.01% ASR on Qwen3-VL-8B, 85.77% on InternVL3.5-8B, 97.95% on Gemma-3-12B-IT (up to +68.36 pp over best baseline). Black-box transfer tests: +24.69 pp on GPT-5.6, +27.34 pp on Gemini 3.1 Pro over FigStep. Multimodal safety alignment remains highly vulnerable to structured visual adversaries.
Aishik Nagar, Cheston Tan — arXiv preprint, Oct 29 2026
Figure 1: SAE feature importance for explicit prompt concepts (turn, occupancy) vs. strategic concepts (symmetries, forks) before and after RL training in Tic-Tac-Toe. RL boosts prominence of input-explicit features but does not install new strategic representations — accuracy gains reflect feature reweighting, not reasoning acquisition.
Uses SAEs and causal interventions to test whether RL training installs genuine game reasoning or reweights existing features. In a fully analyzable Tic-Tac-Toe domain, RL achieves high accuracy and cross-prompt robustness, but SAE analysis reveals the model relies primarily on features already explicit in the prompt (turn, occupancy, game progression); high-level strategic concepts (symmetries, forks) remain weakly represented throughout. RL "reorganizes which features dominate" without adding new concept information. Causal interventions localize decisions to late-layer residual stream states. The finding cautions against interpreting RL benchmark improvements as evidence of reasoning acquisition — a concern for alignment evaluations that use RL-trained models.
Max Neuwinger, Hanna Yukhymenko, Imanol Schlag — arXiv preprint, Oct 29 2026
Figure 1: MSafeTemp pipeline — English harmful prompts are decomposed into slot templates, translated with cultural localization across 60 languages, and filled with locally relevant entities to produce training and evaluation data that exposes safety gaps invisible to plain translation.
Builds MSafeTemp, a 60-language safety training and evaluation corpus generated by extracting slot templates from English harmful prompts and filling them with culturally localized entities. Evaluations show English-only safety evaluation "misjudges multilingual safety": attacks failing in English succeed after translation, and culturally localized variants expose additional failures that plain translation misses. Augmenting safety training with MSafeTemp data improves alignment across model families beyond English. Strengthens the case that safety pipelines and evaluations must be explicitly multilingual; English-only safety scores are insufficient measures of deployed safety.
Chao Chen, Pengyang Shao, Wei Qin, Shijie Hao, Fei Shen, Richang Hong — arXiv preprint, Oct 29 2026
Figure 1: Harmful and over-refusal sample representations separate clearly in intermediate layers but converge in deep layers — explaining why the refusal mechanism over-generalizes to benign queries. GOAL removes over-refusal gradient components along the harmful-gradient low-rank subspace to fix the overlap.
Provides a mechanistic account of over-refusal: harmful and over-refusal samples separate in intermediate layers but overlap in deep layers, and harmful-sample gradients form a low-rank subspace that over-refusal gradients project onto. GOAL (Gradient Orthogonalization with contrAstive Learning) removes over-refusal gradient components along the harmful-gradient subspace and adds a contrastive objective to widen the deep-layer representation gap. Evaluated across multiple models and benchmarks, GOAL mitigates over-refusal while preserving safety. The first low-rank mechanistic account connecting representation geometry to the optimization dynamics that cause over-generalization of the refusal mechanism.