Emergent misalignment — where finetuning on a narrow, benign task causes a model to misbehave on completely unrelated queries — has been documented empirically but never mechanistically explained. This paper gives the first causal explanation: the model's chat-template prefix tokens act as a transmission belt, binding the finetuned behavior to a shared set of keys and values that then appear in every downstream query, benign or not.
Figure 1: Finetuning on a narrow task (e.g. legal QA) binds misaligned behavior to shared chat-template prefix tokens; prefix KV patching with the unfinetuned model's states restores alignment without touching the user query.
Zhao, Wu, Arora, Sun, Bau, and Shi test the piggyback hypothesis across factual QA, abstention, tool-use, and refusal finetuning scenarios. Perturbing the prefix tokens (but not the query tokens) consistently restores alignment; patching the prefix key-value states with those from the unfinetuned model raises Llama-3.1-8B's general alignment from 40.8 to 90.4. Their proposed training intervention, Token-Regularized Finetuning (TReFT), adds a penalty on how far prefix KV states move from their pre-finetuning values. TReFT achieves 33.5% more misalignment reduction than data interleaving on the Llama-3.1-8B legal setting, and reduces off-topic over-generalization by 54.3% on average across four behavior types, while maintaining in-domain task performance (EM-F1 61.4→78.4).
Chain-of-thought monitoring — reading a model's reasoning trace to detect misbehavior — rests on an implicit assumption: if the model's reasoning was necessary for the answer, the trace will reveal what drove that answer. This paper systematically breaks that assumption, showing monitoring failures at all levels of a new four-level CoT necessity hierarchy.
Figure 1 (left): Monitorability scores by necessity level on three hard datasets for Qwen3-8B — failures occur even at the highest necessity level. (right): A medical-diagnosis example where patient sex is causally necessary but never mentioned in the CoT.
Bürger, Komorowski, Weckbecker, Kortukov, Jain, Lapuschkin, and Samek introduce a four-level hierarchy distinguishing when CoT is unnecessary, peripheral, contributory, or necessary for the final answer, and build intervention experiments to place examples at each level. On a hard medical-diagnosis task, patient sex influences at least 19% of correct answers and a clinical attribute influences at least 24% — with a CoT that is necessary for the answer — yet "their influence is almost never articulated" in the reasoning. The finding directly challenges the scalable oversight assumption that monitoring becomes reliable once the model's reasoning is causally connected to its output: necessity is not sufficient for monitorability.
Aligned LLMs owe their refusal behavior not to a diffuse whole-network property but to a sparse set of feed-forward neurons that act jointly — remove as few as 29 of them and safety collapses. CircuitBreak finds this set without needing paired "twin" prompts or per-neuron independent scoring, fixing a blind spot in prior safety-removal attacks.
Figure: CircuitBreak attack success rate on Llama-3.1-8B (StrongREJECT / Llama Guard 3 8B) as a function of neurons removed, showing a rapid rise from 7.35% to 87.54% after removing fewer than 150 neurons (<0.04% of MLPs).
Ying, Wenlin, Yang, and Xiao run a gradient×activation saliency step to rank candidates, then greedily select neurons in order of their joint contribution to flipping the refusal readout, re-scoring every remaining candidate after each removal. Across ten instruction-tuned models from Llama and Qwen families, convergence requires 29–150 neurons (0.007–0.039% of MLP neurons). Llama-3.2-3B's ASR goes from 2.24% to 87.54%; Llama-3.1-8B from 7.35% to 87.54%; Qwen2.5-7B from 8.95% to 88.50%; Qwen2.5-14B from 5.43% to 88.18%. The paper serves as a white-box auditing tool and a call to harden safety against sparse-circuit removal.
Kairos interposes at the OS boundary — agent runs speculatively, deferred effects are held until an oracle (LLM or human with bounded authority) reviews the full trace and releases or refuses them.
Hempel, Loose, Cavallaro, and Eisenbarth place a reference monitor at the OS isolation boundary of an unmodified agent: all tool calls and shell commands funnel through file/network operations, effects are held speculatively, and a bounded-authority oracle releases or refuses them after reviewing the full action trace. On AgentDojo, attack success drops from 47.0% to 2.7% while benign utility is preserved — the largest utility-preserving defence result in the AgentDojo literature. Outperforms existing defences on three benchmarks across three agent models.
Standard probes game position and format cues; matched negatives force the probe to learn injection semantics — cutting false alarm rates by ~10× while maintaining 95% attack recall.
Kale and Vishwanath show that linear probes on hidden states used as injection monitors suffer large false-alarm rates because training data lets them latch onto conversation position and text length rather than injection content. Adding "matched negatives" — injection-free conversations formatted like injections — to training drops false alarms from 41–80% to 4–27% on injection-free traffic and from 92–98% to 3–16% on harmless look-alikes, while raising attack detection from 91% to 95% across six open models.
Providing a codebase consistently reduces measured reward hacking (green); providing benchmark transcripts consistently increases it (red) — a strong configuration artifact that invalidates naive comparisons across audit setups.
Slater, Lundqvist, Richter, and Wei use the Petri tool to audit LLM coding agents across four auditor–target model pairs and three behaviors (reward hacking, harmful compliance, sycophancy). Providing a fixed codebase reduces reward hacking scores by up to 3.5 points; providing benchmark transcripts of the target's past behavior raises reward hacking by up to 4.4 points — even though transcripts might naively seem informative. Careful piloting of auditor configuration is essential before trusting measured behavior rates.
SA-CUA uses the accessibility tree to deterministically classify screen regions as trusted or untrusted; an endorsement channel is the only path for untrusted values to influence the privileged LLM's actions — yielding zero attack success rate.
Seip, Markovinović, and Hein observe that computer-use agents operating on raw pixels cannot draw a clean border between trusted UI and attacker-controlled content. SA-CUA builds a dual-LLM trust architecture that uses the accessibility tree to classify screen regions and introduces an endorsement channel forcing values to cross the trust boundary only via verified paths. On a direct-injection benchmark, SA-CUA achieves zero attack success rate while maintaining substantially higher task utility than the prior CaMeL-CUA defence.
DPOison-VL: imperceptible perturbations computed via a bilevel objective and gradient alignment surrogate invert the preferred/rejected signal for attacker-chosen queries — no changes to prompts, responses, or preference annotations.
Jian and Yang demonstrate the first clean preference poisoning attack on VLM-DPO: imperceptible pixel-level perturbations to training images (no changes to prompts, responses, or labels) invert the DPO preferred/rejected signal for attacker-chosen queries. A bilevel optimization with a gradient alignment surrogate and a preference displacement score for sample selection achieves 98% attack success rate with only 2% poisoned training data, with cross-model transferability across multiple VLM architectures and scales.
JPDH exploits the fact that schema validation cannot detect semantic hijacking — adversarial context shifts the model's structured judgment without triggering any format check, enabling downstream security impact.
Luo, Ji, Kang, Wang, Liu, Yin, and Liu define Jev Program Decision Hijacking (JPDH): adversarial context steers Jev program decisions through structured judgments that still pass schema validation. JevProbe tested 228 real-world Jev applications (2,273 test cases) and found 81.5% exhibiting decision flips, with 98.6% of flips leading to potential downstream security impacts — a 50.4 pp improvement in Decision Flip Rate over the strongest adapted baseline. Existing defences offered limited protection. Responsibly disclosed.
TRAP's three-stage pipeline: strategy tree organises known jailbreak families, a tree-structured bandit policy learns which paths succeed and carries that knowledge across objectives, and RAG-style retrieval instantiates the chosen strategy.
Mirbagheri and Sadeghzadeh build a persistent hierarchy of jailbreak strategies connected to concrete templates, and learn which strategies work via a tree-structured gradient-bandit RL policy that carries preferences across attack objectives. Retrieval-augmented prompt construction uses successful examples for the chosen strategy. Evaluated on AdvBench across seven aligned LLMs, TRAP outperforms baselines on six of seven models with attack success rates of 45.6–97.9% on those six.
