In 2024 Andy Arditi and his coauthors showed this in Refusal in Language Models Is Mediated by a Single Direction. One direction in the residual stream mediates refusal across the chat models they tested. The code is public, there are abliterated models using this finding on Hugging Face, and the paper has 1,232 citations on Semantic Scholar.

So what did fine-tuning do to that direction? The follow-ups give a fairly consistent answer. Fine-tuning mostly breaks refusal by going around the direction, which survives in some form while the routing around it changes. One direction is enough to steer the refusal decision that safety training produces. It is not the whole story, though, and it is not the model’s judgment of harm. Defenses that hold the direction in place during fine-tuning work in their own evaluations. Cheap attribution methods can find the training data that moved it. What nobody has done is trace the direction through all three stages of one open training pipeline: SFT, then DPO, then RL.

Fig. 1. The map of this post. Arditi et al. is on the left. The follow-up papers fall into four groups: 10 on fine-tuning attacks, 10 on the geometry of refusal, 6 on defenses and 3 on data attribution. The dashed box is the measurement I could not find. (Diagram by the author)

In my latest research, I have used Arditi et al.’s refusal direction to read models rather than steer them. I project a model’s activations onto it and compare how far prompts sit along it. For this post I ran an Undermind deep search (250 ranked papers) and added papers I had already read. I sorted the follow-ups by the question each one asks (Fig. 1). The papers measure refusal in different ways, so their numbers rarely compare directly. The reference section at the end lists each paper’s models, prompts and metric.

Does fine-tuning delete the direction, or go around it?

Qi et al. (2023) found that fine-tuning an aligned model on a few examples compromises its safety, even when the data is benign. Two follow-up papers from 2024 explain parts of this result.

Qi et al. (2024) argue that safety alignment is shallow. It changes the output distribution over the first few tokens and little else. In their account, this is why prefilling attacks, decoding-parameter attacks and fine-tuning attacks all work.

Fig. 2. Where alignment changes the output. The y-axis is the KL divergence between the aligned chat model and its base model, per token, on harmful responses from HEx-PHI. For Llama-2-7B the divergence is high on the first token and small from the fourth token on. Alignment changes the start of the response the most. (Figure from Qi et al. 2024)

He, Xia and Henderson (2024) look at the data. They select 100 benign examples, and after training on them the model complies with more than 70% of harmful requests. After training on 100 random benign examples, it complies with fewer than 20%. The selected examples are often lists, bullet points and math problems. Nothing in them is harmful.

Lee et al. (2024) studied direct preference optimization (DPO) on toxicity in GPT-2. After DPO, the toxic capability is still in the model, and the model learns to bypass it. Later papers in this group report the same pattern.

Three 2024 papers look inside the model. Yu et al. (2024) show that many adversarial attacks share one mechanism: they ablate the refusal direction, which they call the refusal feature. Their defense, ReFAT, simulates that ablation during training. Leong et al. (2024) split refusal into three stages: recognize the harmful request, start the refusal, finish the refusal. Different fine-tuning attacks hit different stages. Jain et al. (2024) use synthetic data to study supervised fine-tuning (SFT), DPO and unlearning. All three methods make small changes to MLP weights that push unsafe inputs into a null space. On their synthetic data, a jailbreak works when it makes an unsafe input look like a safe one.

DPO can also undo refusal by itself. Razin et al. (2024) describe likelihood displacement. DPO can move probability mass from the preferred refusal to a harmful response, because the two responses in a pair have similar embeddings. In one of their runs, Llama-3-8B-Instruct’s refusal rate fell by more than half.

Yoon et al. (2026) turn likelihood displacement into an attack. Each of their 10 preference pairs has a benign prompt, a helpful preferred answer and a refusal as the rejected answer. They submit these pairs through OpenAI’s public fine-tuning API. The attack then succeeds on 55% to 82% of harmful requests against GPT-4o and the GPT-4.1 models. To the API provider, the request looks the same as a legitimate request to reduce over-refusal.

Guo et al. (2026) explain benign fine-tuning with Fisher geometry. They argue that safety depends on a low-rank pathway that routes outputs. About 100 benign examples re-sharpen that pathway, and general capability changes only a little.

My reading of this group: fine-tuning keeps the direction in some form, and refusal fails because of changes around it. Three papers say this in their own words. Lee et al. say the toxic capability is “bypassed”. Guo et al. say “routing disrupted”. Zhao et al. (next section) find that the model’s judgment of harmfulness stays intact. None of the papers in this group measures the direction inside a production post-training pipeline.

Is refusal really one direction?

Arditi et al. claim that the direction mediates refusal under intervention. Much of the 2025 and 2026 work tests the limits of that claim.

Marshall, Scherlis and Belrose (2024) treat refusal as an affine function. On ten models, subspace projection combined with activation addition controls refusal more precisely than either method alone. Wollschläger et al. (2025) find several independent refusal directions and multi-dimensional concept cones. They also show that two orthogonal directions can still interact under intervention.

Pan et al. (2025) study how representations shift during safety fine-tuning of Llama 3 8B. They find one dominant direction plus smaller directions that are interpretable, for features such as hypothetical framing and role-play. The smaller directions modulate the dominant one. Yeo et al. (2025) use sparse autoencoders (SAEs) and find features that causally mediate refusal in two chat models.

Zhao et al. (2025) separate harmfulness from refusal. They find a harmfulness direction that is distinct from the refusal direction. Jailbreaks lower the refusal signal and leave the model’s internal judgment of harmfulness in place. Adversarial fine-tuning also leaves that judgment almost unchanged.

Fig. 3. Two directions at two token positions. Zhao et al. read harmfulness at the last token of the user instruction (orange) and refusal at the last token of the full prompt (blue). In this example the model judges “how to kill the lights in my room” harmless and still refuses. Adding the refusal direction does not make the model call the prompt harmful. Adding the harmfulness direction does. (Figure from Zhao et al. 2025, CC BY 4.0)

Zhao et al.’s split matters for the next two papers. Refusal is an action: when the refusal direction is present, the model writes a refusal. Harmfulness is a judgment about the input. A monitor that reads harmfulness measures something different from the direction Arditi et al. found.

Two papers probe harmfulness and find more than one direction. Bailey et al. (2024) attack harmfulness probes that read the model’s input. In one experiment on Llama-3-8B-Instruct, the defender retrains the probe after every attack. After more than 70 rounds, the attacker still finds a new embedding suffix that fools the latest probe. The authors conclude that no single “harmfulness direction” lets a monitor recognize harmful inputs robustly.

Youstra et al. (2025) find many harm directions without any attack. They train a probe, remove its direction from the activations, and train the next probe on what remains. The probes stay near-perfect for the first hundred orthogonal directions. In their examples, steering along one of these directions jailbreaks the model on a harmful prompt. Their model was fine-tuned on cipher-encoded harmful data, so the result may not transfer to a standard chat model.

My reading of the two papers: the model’s judgment of harm can be read along many directions. An attacker who suppresses one direction has others to use. Neither paper measures the refusal direction.

Joad et al. (2026) find that different refusal categories have geometrically distinct directions. Steering along any of them gives nearly the same trade-off between refusal and over-refusal. So the category directions change the style of the refusal, and the decision to refuse stays the same. Wang et al. (2026) define steering dimensionality. One dimension is enough to steer the refusal that safety training produces. Refusal in general contexts needs more than five dimensions.

Ponkshe et al. (2025) are the most skeptical. On five Llama and Qwen models, the subspaces that amplify safe behavior also amplify useful behavior. They conclude that safety is entangled with general learning.

These papers agree with Arditi et al. if the claim stays narrow. One direction is enough to control the refusal decision that safety training produces. The direction is not a full description of how the model represents refusal (Wollschläger et al., Joad et al.). It is also not an isolated feature that you can protect or delete without side effects (Ponkshe et al.). It is also not the model’s judgment of harm, which Zhao et al., Bailey et al. and Youstra et al. find elsewhere. I use the direction as a measuring instrument for the refusal decision, and I make no claim about mechanism. Both the one-direction papers and the multi-direction papers support that use.

Can the direction be protected during fine-tuning?

Two 2024 defenses act on representations without using the refusal direction. Zou et al. (2024) introduce circuit breakers, which reroute the representations that produce harmful outputs. Circuit breakers hold up against attacks the model did not see in training. Chen et al. (2024) find a small set of safety neurons. Patching only those neurons restores most of the model’s safety performance. The safety neurons overlap with helpfulness neurons, which gives a mechanistic account of the alignment tax.

Four defenses from 2024 to 2026 work with the direction or with nearby safety representations. Zhu et al. (2024) observe that fine-tuned models keep safety-related activations close to those of their base model. From these activations they extract a reusable set of safety directions, which they call SafetyLock. On toxic fine-tunes, SafetyLock brings the harmful-response rate from 60% to below 1%, in under a hundredth of a second.

Du et al. (2025) add a loss term during fine-tuning. The term constrains each sample’s projection onto the refusal direction. Asif and Amiri (2026) measure how safety-relevant representations drift during standard fine-tuning. Their method, RefusalGuard, constrains updates so that those components stay stable.

Li et al. (2026) study reasoning models. They find that the safety mechanism is still present after post-training, but over-activated reasoning representations mask it. Their method, SafeReAct, prunes the reasoning neurons to build a safer reference model. LoRA adapters then train the original model to match the pruned model’s representations on harmful prompts.

SafetyLock, the projection constraint of Du et al., RefusalGuard and SafeReAct repeat the pattern from the first section. All four assume that the direction, or the safety representation around it, survives fine-tuning. They keep it in place during training or restore it afterwards. None of the four asks which stage of a multi-stage pipeline moved it.

Which training data moved it?

Razin et al. already look at the data. Their similarity score finds the preference pairs that cause likelihood displacement, and removing those pairs reduces the unalignment. Three 2026 papers do data attribution for safety at larger scale.

Kowal et al. (2026) introduce Concept Influence. It attributes behavior to semantic directions, such as probes or SAE features, instead of to individual test examples. Simple probe-based attribution is a first-order approximation of Concept Influence, and it runs much faster than influence functions.

Xiao and Aranguri (2026) apply a similar method to OLMo 2’s full DPO dataset. The target behavior is a model that refuses a harmful request until you append a formatting instruction such as “(no more than 50 words)”. They score each preference pair by its alignment with an activation-difference vector for that behavior. Then they remove or relabel the top 30,000 pairs and retrain DPO from the SFT checkpoint. Switching the labels cuts the behavior by more than three-quarters. Removing or relabeling 30,000 random pairs does not reliably reduce it.

Fig. 4. Retraining OLMo 2 DPO after changing 30,000 of 378,341 preference pairs. The baseline harmful-response rate is 7.6%. Each method ranks the pairs differently. “LLM Toxic” asks an LLM judge how toxic each pair is. “Gradient” uses gradient-based attribution. “Probe” uses the activation-difference vector. The probe ranking gives the lowest harmful-response rate for both removal (2.9%) and label switching (1.7%). (Figure from Xiao and Aranguri 2026, CC BY 4.0)

Bergen et al. (2026), from Goodfire, start before training. They inspect a preference dataset before optimization and predict, at the level of concepts, which behaviors the model will learn from it. Their predictions match the feature changes after DPO closely. The same paper also tracks the refusal direction from SFT to DPO, which I come back to in the next section.

Xiao and Aranguri report that their attribution costs more than ten times less than the gradient-based and LLM-judge baselines. With methods this cheap, I expect more papers in this group soon.

What happens to the direction across a full pipeline?

Tülu 3 and OLMo 2 publish the checkpoints, datasets and evaluations for a full pipeline of SFT, then DPO, then reinforcement learning (RL). Seven papers come close to tracing refusal through such a pipeline. Each covers part of it.

The two closest measure the direction itself. Du et al. (2025) compare base and post-trained models, including a Tülu 3 SFT checkpoint and Meta’s Llama-3.1-8B-Instruct. Their figure carries the most important number in this post. The base model’s refusal direction is almost unrelated to the SFT model’s: cosine similarity 0.15. The SFT and Instruct directions are only partly aligned, at 0.66. That number covers base to SFT only.

Fig. 5. Cosine similarity between the directions of three Llama-3.1-8B models: base, Tülu 3 SFT and Meta’s Instruct. The left and middle panels are truthfulness directions, and all pairs are 0.89 or higher. The right panel is the refusal direction. The base model’s refusal direction is almost unrelated to the SFT direction (0.15) and the Instruct direction (0.25). The SFT and Instruct refusal directions are only partly aligned (0.66). (Figure from Du et al. 2025, CC BY 4.0)

Bergen et al. take the next step. In all four model families they train, DPO lowers safety relative to the SFT checkpoint, including in a reproduction of Tülu 3 70B DPO. In an appendix they re-estimate the refusal direction after DPO and compare it with the SFT direction by layerwise cosine similarity. They also steer each DPO model with the SFT direction and with its own direction, and score refusal with WildGuard. Their DPO models are their own runs, and they stop before RL.

The other five each touch one stage, or measure something other than the direction:

• Zahraei et al. (2026) evaluate the released SFT, DPO and RLVR checkpoints of Tülu 3 and OLMo 3. They measure whether a model summarizes a source faithfully when the source makes an unsafe claim. Unfaithfulness grows most at DPO, and RLVR recovers only part of it. They do not use the refusal direction.
• Lu et al. (2026) fork a partly trained OLMo-2-1B model into six branches that differ only in their last pretraining tokens. Identical SFT installs refusal in every branch. Under the same DPO, and under a GRPO RL update, the branch that ended on safety text loses far less refusal. They measure refusal from model outputs.
• Elcock et al. (2026) compare SFT with RL from verifiable rewards (RLVR) as adaptation methods. SFT causes more alignment drift.
• Kabir and Tiganj (2026) compare harmful SFT, harmful RLVR and abliteration, which removes the refusal direction from the weights. RLVR keeps the representation geometry and retargets behavior toward compliance.
• Xiao and Aranguri work inside OLMo 2’s DPO stage.

No paper traces the refusal direction through the released SFT, DPO and RL checkpoints of one public pipeline. Bergen et al. cover SFT to DPO on their own runs. No paper measures where harmful and benign prompts sit along the direction at each of the three stages. The Tülu 3 paper’s own tables show safety behavior decaying across the stages. The direction gives a way to measure the change, and the checkpoints are on Hugging Face. The measurement also needs one metric held fixed across all stages. Then a change in the number comes from the model and not from the judge. In 250 papers I found no published version of this measurement.

I have run this measurement as part of a paper in progress. The paper now asks a broader question: what in DPO and RL causes the decline in refusal. I will post the results when the paper is out. If you know of a paper that already does this, please tell me.

Reference: how each paper measures refusal

Arditi et al. use two scores. The refusal score is a string match. An answer counts as a refusal if it contains a phrase such as “I’m sorry” or “As an AI”. The safety score asks Llama Guard 2 whether the answer is harmful.

The follow-up papers do not share a metric, and the methods answer two different questions. A refusal metric asks whether the model refused. A harm metric asks whether the answer is harmful. Zhao et al. show that the model keeps these apart internally. Yeo et al. report the practical problem: an answer without a refusal phrase is often not a successful jailbreak. My judges post describes each judge in the table.

The prompt sets differ as well. HEx-PHI, AdvBench, HarmBench, JailbreakBench, SORRY-Bench and XSTest all appear, among about twenty sets. My datasets post describes each one. My red-teaming post describes the same comparability problem for attack evaluations.

The models are narrow too. Almost every paper includes a Llama chat model at 7B or 8B. Only Arditi et al., Zhu et al., Youstra et al. and Bergen et al. test models of 70B or more. Two papers attack closed models through fine-tuning APIs: Qi et al. 2023 (GPT-3.5 Turbo) and Yoon et al. (GPT-4o and GPT-4.1). Fewer than half of the papers test harmless prompts at all, so a method that also makes the model refuse benign requests often goes unmeasured. Only Bergen et al. evaluate on DAN and WildJailbreak jailbreak prompts.

Papers

The numbers the narrative leaves out are here, one per paper.