Published:
The bug was one line of shell. The eval script found the last checkpoint with sort -t- -k2 -n, which splits a path on hyphens. The project lives in a folder called safety-decay, so the sort keyed on a field that never changed. It fell back to alphabetical order, where checkpoint-9372 comes after checkpoint-16875. The agent evaluated a checkpoint that was 55% of the way through training.
Published:
People like to call prompt injection the SQL injection of LLMs, and the comparison holds up better than most. Untrusted input and trusted instructions travel down the same channel, and the model has no built-in way to tell them apart. I find it useful to go back to what this looked like in 2022, when the models were GPT-3 variants and the attacks were brand new, and put it next to where we are now.
Published:
The bug was one line of shell. The eval script found the last checkpoint with sort -t- -k2 -n, which splits a path on hyphens. The project lives in a folder called safety-decay, so the sort keyed on a field that never changed. It fell back to alphabetical order, where checkpoint-9372 comes after checkpoint-16875. The agent evaluated a checkpoint that was 55% of the way through training.
Published:
When Copilot and Codex showed up in everyone’s editor, the security community had a fast and fair reaction. A model trained on a huge pile of public code is going to suggest some of the bad patterns in that pile, and people are going to accept them. There was even evidence for the worry. When you looked at what Copilot generated on its own, a meaningful fraction of it contained weaknesses (Pearce et al. 2022).
Published:
Xiao and Aranguri found this failure in OLMo 2, Ai2’s open model, and call it distractor-triggered compliance. It raises a concrete question: which training examples taught the model to do that?
Published:
Most applications built on a language model work the same way. You write an instruction, append the user’s text, and send the whole thing to the model:
Published:
In August 2024, TAR (Tamper-Resistant Safeguards) arrived with the strongest tamper-resistance claim anyone had published: safeguards on an open-weight model that survived up to 5,000 steps of adversarial fine-tuning (Tamirisa et al. 2024, v1). Four months later, Qi, Wei, Carlini and colleagues changed two hyperparameters. One hundred warmup steps instead of thirty, cosine learning-rate decay instead of a constant rate after warmup. Post-attack WMDP-Bio accuracy went from roughly 25%, which is chance, to consistently above 60%, near the undefended Llama-3-8B-Instruct baseline (Qi et al. 2024). TAR’s current arXiv version says “hundreds of steps.”
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
An attack success rate (ASR) is the share of harmful prompts where a judge says the attack worked. So every ASR is a judge’s verdict. Judges ask one of two questions: did the model refuse, or is the answer harmful? On the same answers, the two questions give different numbers. Souly et al. (2024) find that most earlier automated judges overstate jailbreak success compared with human ratings. Refusal-phrase matching overstates it most.
Published:
A refusal rate or attack success rate (ASR) from one paper compares with another only when both use the same prompt set, split and judge. ASR is the share of harmful prompts where the judge says the attack worked. A split is a fixed subset of a set, such as a test split or a public release. HEx-PHI and AdvBench both hold harmful requests. They differ in categories and wording, so one model gets a different refusal rate on each. An AdvBench result also says nothing about XSTest, where the prompts are harmless.
Published:
In August 2024, TAR (Tamper-Resistant Safeguards) arrived with the strongest tamper-resistance claim anyone had published: safeguards on an open-weight model that survived up to 5,000 steps of adversarial fine-tuning (Tamirisa et al. 2024, v1). Four months later, Qi, Wei, Carlini and colleagues changed two hyperparameters. One hundred warmup steps instead of thirty, cosine learning-rate decay instead of a constant rate after warmup. Post-attack WMDP-Bio accuracy went from roughly 25%, which is chance, to consistently above 60%, near the undefended Llama-3-8B-Instruct baseline (Qi et al. 2024). TAR’s current arXiv version says “hundreds of steps.”
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Xiao and Aranguri found this failure in OLMo 2, Ai2’s open model, and call it distractor-triggered compliance. It raises a concrete question: which training examples taught the model to do that?
Published:
In 2024 Andy Arditi and his coauthors showed this in Refusal in Language Models Is Mediated by a Single Direction. One direction in the residual stream mediates refusal across the chat models they tested. The code is public, there are abliterated models using this finding on Hugging Face, and the paper has 1,232 citations on Semantic Scholar.
Published:
The model is Llama-3.1-8B-Instruct. It flags sprintf in 90% of code reviews and writes sprintf in 93.3% of generation trials. This post is the long version of our preprint Surgical Repair of Insecure Code Generation in LLMs (Sandoval, Dolan-Gavitt and Garg 2026). I wrote it for an alignment audience. Many alignment proposals assume that a model acts on what it can state. Here it does not, and the failure is cheap to measure. A short summary from June already exists.
Published:
I have to confess something: I have been vibe coding and beyond for over a couple of years now. I started with CoPilot, for which I even created a plugin for a research paper (https://www.usenix.org/conference/usenixsecurity23/presentation/sandoval), then moved on to Cursor and eventually landed in Claude Code. I have created my initial Claude.md for research that has different personas, etc. However, given that I have spent the past (few) years in Academia, this has mainly been either a solo journey or with a few graduate students.
Published:
The two models are Marin-8B-Instruct and Olmo-3-7B-Instruct. The attack is a LoRA fine-tune, which trains a small set of added weights, on about 100 public examples. By step 20, both models reach 99.1% attack success on HarmBench, the share of answers a judge labels harmful. The pretraining result comes from Marin’s base model. Its willingness to write misinformation is lowest at Phoenix, the phase that adds web data, at 51.7%. After the final cooldown, a last stretch of training on a curated mix, it reaches 73.9%. Most of that rise comes from the model refusing less and endorsing the false claim more. A third finding limits the first two. My main judge, WildGuard, scores the side an answer takes and almost ignores how well it is written. So several of my numbers measure how often the model took the requested side. That is narrower than how harmful the text was.
Published:
In August 2024, TAR (Tamper-Resistant Safeguards) arrived with the strongest tamper-resistance claim anyone had published: safeguards on an open-weight model that survived up to 5,000 steps of adversarial fine-tuning (Tamirisa et al. 2024, v1). Four months later, Qi, Wei, Carlini and colleagues changed two hyperparameters. One hundred warmup steps instead of thirty, cosine learning-rate decay instead of a constant rate after warmup. Post-attack WMDP-Bio accuracy went from roughly 25%, which is chance, to consistently above 60%, near the undefended Llama-3-8B-Instruct baseline (Qi et al. 2024). TAR’s current arXiv version says “hundreds of steps.”
Published:
In May 2024, Anthropic put out a version of Claude that could not stop talking about the Golden Gate Bridge. Ask it for a banana bread recipe and it would work the bridge into the ingredients. Ask it to write a business email and the fog would roll in. It was funny, it was briefly a meme, and it was made by turning a single knob. The researchers had located an internal feature that fired on the concept “Golden Gate Bridge,” clamped its value high, and let the model run (Templeton et al. 2024).
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
People like to call prompt injection the SQL injection of LLMs, and the comparison holds up better than most. Untrusted input and trusted instructions travel down the same channel, and the model has no built-in way to tell them apart. I find it useful to go back to what this looked like in 2022, when the models were GPT-3 variants and the attacks were brand new, and put it next to where we are now.
Published:
Ask an LLM to generate a piece of code and it might hand you a SQL injection or a buffer overflow. Ask the same model, a minute later, “does this code have a vulnerability?” and it will often name the bug, explain why it’s dangerous, and write the fix. So the knowledge is in there. The model just doesn’t use it while it’s generating.
Published:
When Copilot and Codex showed up in everyone’s editor, the security community had a fast and fair reaction. A model trained on a huge pile of public code is going to suggest some of the bad patterns in that pile, and people are going to accept them. There was even evidence for the worry. When you looked at what Copilot generated on its own, a meaningful fraction of it contained weaknesses (Pearce et al. 2022).
Published:
An attack success rate (ASR) is the share of harmful prompts where a judge says the attack worked. So every ASR is a judge’s verdict. Judges ask one of two questions: did the model refuse, or is the answer harmful? On the same answers, the two questions give different numbers. Souly et al. (2024) find that most earlier automated judges overstate jailbreak success compared with human ratings. Refusal-phrase matching overstates it most.
Published:
A refusal rate or attack success rate (ASR) from one paper compares with another only when both use the same prompt set, split and judge. ASR is the share of harmful prompts where the judge says the attack worked. A split is a fixed subset of a set, such as a test split or a public release. HEx-PHI and AdvBench both hold harmful requests. They differ in categories and wording, so one model gets a different refusal rate on each. An AdvBench result also says nothing about XSTest, where the prompts are harmless.
Published:
Most applications built on a language model work the same way. You write an instruction, append the user’s text, and send the whole thing to the model:
Published:
Xiao and Aranguri found this failure in OLMo 2, Ai2’s open model, and call it distractor-triggered compliance. It raises a concrete question: which training examples taught the model to do that?
Published:
In 2024 Andy Arditi and his coauthors showed this in Refusal in Language Models Is Mediated by a Single Direction. One direction in the residual stream mediates refusal across the chat models they tested. The code is public, there are abliterated models using this finding on Hugging Face, and the paper has 1,232 citations on Semantic Scholar.
Published:
The model is Llama-3.1-8B-Instruct. It flags sprintf in 90% of code reviews and writes sprintf in 93.3% of generation trials. This post is the long version of our preprint Surgical Repair of Insecure Code Generation in LLMs (Sandoval, Dolan-Gavitt and Garg 2026). I wrote it for an alignment audience. Many alignment proposals assume that a model acts on what it can state. Here it does not, and the failure is cheap to measure. A short summary from June already exists.
Published:
In May 2024, Anthropic put out a version of Claude that could not stop talking about the Golden Gate Bridge. Ask it for a banana bread recipe and it would work the bridge into the ingredients. Ask it to write a business email and the fog would roll in. It was funny, it was briefly a meme, and it was made by turning a single knob. The researchers had located an internal feature that fired on the concept “Golden Gate Bridge,” clamped its value high, and let the model run (Templeton et al. 2024).
Published:
Ask an LLM to generate a piece of code and it might hand you a SQL injection or a buffer overflow. Ask the same model, a minute later, “does this code have a vulnerability?” and it will often name the bug, explain why it’s dangerous, and write the fix. So the knowledge is in there. The model just doesn’t use it while it’s generating.
Published:
The two models are Marin-8B-Instruct and Olmo-3-7B-Instruct. The attack is a LoRA fine-tune, which trains a small set of added weights, on about 100 public examples. By step 20, both models reach 99.1% attack success on HarmBench, the share of answers a judge labels harmful. The pretraining result comes from Marin’s base model. Its willingness to write misinformation is lowest at Phoenix, the phase that adds web data, at 51.7%. After the final cooldown, a last stretch of training on a curated mix, it reaches 73.9%. Most of that rise comes from the model refusing less and endorsing the false claim more. A third finding limits the first two. My main judge, WildGuard, scores the side an answer takes and almost ignores how well it is written. So several of my numbers measure how often the model took the requested side. That is narrower than how harmful the text was.
Published:
In August 2024, TAR (Tamper-Resistant Safeguards) arrived with the strongest tamper-resistance claim anyone had published: safeguards on an open-weight model that survived up to 5,000 steps of adversarial fine-tuning (Tamirisa et al. 2024, v1). Four months later, Qi, Wei, Carlini and colleagues changed two hyperparameters. One hundred warmup steps instead of thirty, cosine learning-rate decay instead of a constant rate after warmup. Post-attack WMDP-Bio accuracy went from roughly 25%, which is chance, to consistently above 60%, near the undefended Llama-3-8B-Instruct baseline (Qi et al. 2024). TAR’s current arXiv version says “hundreds of steps.”
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Xiao and Aranguri found this failure in OLMo 2, Ai2’s open model, and call it distractor-triggered compliance. It raises a concrete question: which training examples taught the model to do that?
Published:
In 2024 Andy Arditi and his coauthors showed this in Refusal in Language Models Is Mediated by a Single Direction. One direction in the residual stream mediates refusal across the chat models they tested. The code is public, there are abliterated models using this finding on Hugging Face, and the paper has 1,232 citations on Semantic Scholar.
Published:
People like to call prompt injection the SQL injection of LLMs, and the comparison holds up better than most. Untrusted input and trusted instructions travel down the same channel, and the model has no built-in way to tell them apart. I find it useful to go back to what this looked like in 2022, when the models were GPT-3 variants and the attacks were brand new, and put it next to where we are now.
Published:
The two models are Marin-8B-Instruct and Olmo-3-7B-Instruct. The attack is a LoRA fine-tune, which trains a small set of added weights, on about 100 public examples. By step 20, both models reach 99.1% attack success on HarmBench, the share of answers a judge labels harmful. The pretraining result comes from Marin’s base model. Its willingness to write misinformation is lowest at Phoenix, the phase that adds web data, at 51.7%. After the final cooldown, a last stretch of training on a curated mix, it reaches 73.9%. Most of that rise comes from the model refusing less and endorsing the false claim more. A third finding limits the first two. My main judge, WildGuard, scores the side an answer takes and almost ignores how well it is written. So several of my numbers measure how often the model took the requested side. That is narrower than how harmful the text was.
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
An attack success rate (ASR) is the share of harmful prompts where a judge says the attack worked. So every ASR is a judge’s verdict. Judges ask one of two questions: did the model refuse, or is the answer harmful? On the same answers, the two questions give different numbers. Souly et al. (2024) find that most earlier automated judges overstate jailbreak success compared with human ratings. Refusal-phrase matching overstates it most.
Published:
A refusal rate or attack success rate (ASR) from one paper compares with another only when both use the same prompt set, split and judge. ASR is the share of harmful prompts where the judge says the attack worked. A split is a fixed subset of a set, such as a test split or a public release. HEx-PHI and AdvBench both hold harmful requests. They differ in categories and wording, so one model gets a different refusal rate on each. An AdvBench result also says nothing about XSTest, where the prompts are harmless.
Published:
The bug was one line of shell. The eval script found the last checkpoint with sort -t- -k2 -n, which splits a path on hyphens. The project lives in a folder called safety-decay, so the sort keyed on a field that never changed. It fell back to alphabetical order, where checkpoint-9372 comes after checkpoint-16875. The agent evaluated a checkpoint that was 55% of the way through training.
Published:
Xiao and Aranguri found this failure in OLMo 2, Ai2’s open model, and call it distractor-triggered compliance. It raises a concrete question: which training examples taught the model to do that?
Published:
In 2024 Andy Arditi and his coauthors showed this in Refusal in Language Models Is Mediated by a Single Direction. One direction in the residual stream mediates refusal across the chat models they tested. The code is public, there are abliterated models using this finding on Hugging Face, and the paper has 1,232 citations on Semantic Scholar.
Published:
The two models are Marin-8B-Instruct and Olmo-3-7B-Instruct. The attack is a LoRA fine-tune, which trains a small set of added weights, on about 100 public examples. By step 20, both models reach 99.1% attack success on HarmBench, the share of answers a judge labels harmful. The pretraining result comes from Marin’s base model. Its willingness to write misinformation is lowest at Phoenix, the phase that adds web data, at 51.7%. After the final cooldown, a last stretch of training on a curated mix, it reaches 73.9%. Most of that rise comes from the model refusing less and endorsing the false claim more. A third finding limits the first two. My main judge, WildGuard, scores the side an answer takes and almost ignores how well it is written. So several of my numbers measure how often the model took the requested side. That is narrower than how harmful the text was.
Published:
An attack success rate (ASR) is the share of harmful prompts where a judge says the attack worked. So every ASR is a judge’s verdict. Judges ask one of two questions: did the model refuse, or is the answer harmful? On the same answers, the two questions give different numbers. Souly et al. (2024) find that most earlier automated judges overstate jailbreak success compared with human ratings. Refusal-phrase matching overstates it most.
Published:
A refusal rate or attack success rate (ASR) from one paper compares with another only when both use the same prompt set, split and judge. ASR is the share of harmful prompts where the judge says the attack worked. A split is a fixed subset of a set, such as a test split or a public release. HEx-PHI and AdvBench both hold harmful requests. They differ in categories and wording, so one model gets a different refusal rate on each. An AdvBench result also says nothing about XSTest, where the prompts are harmless.
Published:
The model is Llama-3.1-8B-Instruct. It flags sprintf in 90% of code reviews and writes sprintf in 93.3% of generation trials. This post is the long version of our preprint Surgical Repair of Insecure Code Generation in LLMs (Sandoval, Dolan-Gavitt and Garg 2026). I wrote it for an alignment audience. Many alignment proposals assume that a model acts on what it can state. Here it does not, and the failure is cheap to measure. A short summary from June already exists.
Published:
Ask an LLM to generate a piece of code and it might hand you a SQL injection or a buffer overflow. Ask the same model, a minute later, “does this code have a vulnerability?” and it will often name the bug, explain why it’s dangerous, and write the fix. So the knowledge is in there. The model just doesn’t use it while it’s generating.
Published:
The model is Llama-3.1-8B-Instruct. It flags sprintf in 90% of code reviews and writes sprintf in 93.3% of generation trials. This post is the long version of our preprint Surgical Repair of Insecure Code Generation in LLMs (Sandoval, Dolan-Gavitt and Garg 2026). I wrote it for an alignment audience. Many alignment proposals assume that a model acts on what it can state. Here it does not, and the failure is cheap to measure. A short summary from June already exists.
Published:
I have to confess something: I have been vibe coding and beyond for over a couple of years now. I started with CoPilot, for which I even created a plugin for a research paper (https://www.usenix.org/conference/usenixsecurity23/presentation/sandoval), then moved on to Cursor and eventually landed in Claude Code. I have created my initial Claude.md for research that has different personas, etc. However, given that I have spent the past (few) years in Academia, this has mainly been either a solo journey or with a few graduate students.
Published:
In August 2024, TAR (Tamper-Resistant Safeguards) arrived with the strongest tamper-resistance claim anyone had published: safeguards on an open-weight model that survived up to 5,000 steps of adversarial fine-tuning (Tamirisa et al. 2024, v1). Four months later, Qi, Wei, Carlini and colleagues changed two hyperparameters. One hundred warmup steps instead of thirty, cosine learning-rate decay instead of a constant rate after warmup. Post-attack WMDP-Bio accuracy went from roughly 25%, which is chance, to consistently above 60%, near the undefended Llama-3-8B-Instruct baseline (Qi et al. 2024). TAR’s current arXiv version says “hundreds of steps.”
Published:
In May 2024, Anthropic put out a version of Claude that could not stop talking about the Golden Gate Bridge. Ask it for a banana bread recipe and it would work the bridge into the ingredients. Ask it to write a business email and the fog would roll in. It was funny, it was briefly a meme, and it was made by turning a single knob. The researchers had located an internal feature that fired on the concept “Golden Gate Bridge,” clamped its value high, and let the model run (Templeton et al. 2024).
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
Series: Part 1: Process & Measurement · Part 2: What Holds on Defense · Part 3: Open Weights
Published:
People like to call prompt injection the SQL injection of LLMs, and the comparison holds up better than most. Untrusted input and trusted instructions travel down the same channel, and the model has no built-in way to tell them apart. I find it useful to go back to what this looked like in 2022, when the models were GPT-3 variants and the attacks were brand new, and put it next to where we are now.
Published:
Ask an LLM to generate a piece of code and it might hand you a SQL injection or a buffer overflow. Ask the same model, a minute later, “does this code have a vulnerability?” and it will often name the bug, explain why it’s dangerous, and write the fix. So the knowledge is in there. The model just doesn’t use it while it’s generating.
Published:
When Copilot and Codex showed up in everyone’s editor, the security community had a fast and fair reaction. A model trained on a huge pile of public code is going to suggest some of the bad patterns in that pile, and people are going to accept them. There was even evidence for the worry. When you looked at what Copilot generated on its own, a meaningful fraction of it contained weaknesses (Pearce et al. 2022).
Published:
Most applications built on a language model work the same way. You write an instruction, append the user’s text, and send the whole thing to the model:
Published:
The model is Llama-3.1-8B-Instruct. It flags sprintf in 90% of code reviews and writes sprintf in 93.3% of generation trials. This post is the long version of our preprint Surgical Repair of Insecure Code Generation in LLMs (Sandoval, Dolan-Gavitt and Garg 2026). I wrote it for an alignment audience. Many alignment proposals assume that a model acts on what it can state. Here it does not, and the failure is cheap to measure. A short summary from June already exists.
Published:
In May 2024, Anthropic put out a version of Claude that could not stop talking about the Golden Gate Bridge. Ask it for a banana bread recipe and it would work the bridge into the ingredients. Ask it to write a business email and the fog would roll in. It was funny, it was briefly a meme, and it was made by turning a single knob. The researchers had located an internal feature that fired on the concept “Golden Gate Bridge,” clamped its value high, and let the model run (Templeton et al. 2024).
Published:
In August 2024, TAR (Tamper-Resistant Safeguards) arrived with the strongest tamper-resistance claim anyone had published: safeguards on an open-weight model that survived up to 5,000 steps of adversarial fine-tuning (Tamirisa et al. 2024, v1). Four months later, Qi, Wei, Carlini and colleagues changed two hyperparameters. One hundred warmup steps instead of thirty, cosine learning-rate decay instead of a constant rate after warmup. Post-attack WMDP-Bio accuracy went from roughly 25%, which is chance, to consistently above 60%, near the undefended Llama-3-8B-Instruct baseline (Qi et al. 2024). TAR’s current arXiv version says “hundreds of steps.”
Published:
When Copilot and Codex showed up in everyone’s editor, the security community had a fast and fair reaction. A model trained on a huge pile of public code is going to suggest some of the bad patterns in that pile, and people are going to accept them. There was even evidence for the worry. When you looked at what Copilot generated on its own, a meaningful fraction of it contained weaknesses (Pearce et al. 2022).
Published:
I have to confess something: I have been vibe coding and beyond for over a couple of years now. I started with CoPilot, for which I even created a plugin for a research paper (https://www.usenix.org/conference/usenixsecurity23/presentation/sandoval), then moved on to Cursor and eventually landed in Claude Code. I have created my initial Claude.md for research that has different personas, etc. However, given that I have spent the past (few) years in Academia, this has mainly been either a solo journey or with a few graduate students.