Gustavo Sandoval

Gustavo Sandoval

Sitemap

A list of all the posts and pages found on the site. For you robots out there is an XML version available for digesting as well.

Pages

Posts

The rules my research agents actually follow

Published:

The bug was one line of shell. The eval script found the last checkpoint with sort -t- -k2 -n, which splits a path on hyphens. The project lives in a folder called safety-decay, so the sort keyed on a field that never changed. It fell back to alphabetical order, where checkpoint-9372 comes after checkpoint-16875. The agent evaluated a checkpoint that was 55% of the way through training.

Models Know the Fix and Do Not Use It: A Late-Layer Account of Insecure Code Generation

Published:

The model is Llama-3.1-8B-Instruct. It flags sprintf in 90% of code reviews and writes sprintf in 93.3% of generation trials. This post is the long version of our preprint Surgical Repair of Insecure Code Generation in LLMs (Sandoval, Dolan-Gavitt and Garg 2026). I wrote it for an alignment audience. Many alignment proposals assume that a model acts on what it can state. Here it does not, and the failure is cheap to measure. A short summary from June already exists.

AI Literacy Audit and Rollout Plan

Published:

I have to confess something: I have been vibe coding and beyond for over a couple of years now. I started with CoPilot, for which I even created a plugin for a research paper (https://www.usenix.org/conference/usenixsecurity23/presentation/sandoval), then moved on to Cursor and eventually landed in Claude Code. I have created my initial Claude.md for research that has different personas, etc. However, given that I have spent the past (few) years in Academia, this has mainly been either a solo journey or with a few graduate students.

Red-Teaming Marin: What Fine-Tuning Removes and What Pretraining Adds

Published:

The two models are Marin-8B-Instruct and Olmo-3-7B-Instruct. The attack is a LoRA fine-tune, which trains a small set of added weights, on about 100 public examples. By step 20, both models reach 99.1% attack success on HarmBench, the share of answers a judge labels harmful. The pretraining result comes from Marin’s base model. Its willingness to write misinformation is lowest at Phoenix, the phase that adds web data, at 51.7%. After the final cooldown, a last stretch of training on a curated mix, it reaches 73.9%. Most of that rise comes from the model refusing less and endorsing the false claim more. A third finding limits the first two. My main judge, WildGuard, scores the side an answer takes and almost ignores how well it is written. So several of my numbers measure how often the model took the requested side. That is narrower than how harmful the text was.

What Counts as an Adequate Threat Model?

Published:

In August 2024, TAR (Tamper-Resistant Safeguards) arrived with the strongest tamper-resistance claim anyone had published: safeguards on an open-weight model that survived up to 5,000 steps of adversarial fine-tuning (Tamirisa et al. 2024, v1). Four months later, Qi, Wei, Carlini and colleagues changed two hyperparameters. One hundred warmup steps instead of thirty, cosine learning-rate decay instead of a constant rate after warmup. Post-attack WMDP-Bio accuracy went from roughly 25%, which is chance, to consistently above 60%, near the undefended Llama-3-8B-Instruct baseline (Qi et al. 2024). TAR’s current arXiv version says “hundreds of steps.”

Mechanistic Interpretability as a Security Tool

Published:

In May 2024, Anthropic put out a version of Claude that could not stop talking about the Golden Gate Bridge. Ask it for a banana bread recipe and it would work the bridge into the ingredients. Ask it to write a business email and the fog would roll in. It was funny, it was briefly a meme, and it was made by turning a single knob. The researchers had located an internal feature that fired on the concept “Golden Gate Bridge,” clamped its value high, and let the model run (Templeton et al. 2024).

Prompt Injection, 2022 vs Today: A Retrospective

Published:

People like to call prompt injection the SQL injection of LLMs, and the comparison holds up better than most. Untrusted input and trusted instructions travel down the same channel, and the model has no built-in way to tell them apart. I find it useful to go back to what this looked like in 2022, when the models were GPT-3 variants and the attacks were brand new, and put it next to where we are now.

The Format-Reliability Gap: Diagnosing and Repairing Insecure Code Generation

Published:

Ask an LLM to generate a piece of code and it might hand you a SQL injection or a buffer overflow. Ask the same model, a minute later, “does this code have a vulnerability?” and it will often name the bug, explain why it’s dangerous, and write the fix. So the knowledge is in there. The model just doesn’t use it while it’s generating.

Does AI Make You Write Insecure Code? A User Study

Published:

When Copilot and Codex showed up in everyone’s editor, the security community had a fast and fair reaction. A model trained on a huge pile of public code is going to suggest some of the bad patterns in that pile, and people are going to accept them. There was even evidence for the worry. When you looked at what Copilot generated on its own, a meaningful fraction of it contained weaknesses (Pearce et al. 2022).

The Judges Behind LLM Safety Numbers

Published:

An attack success rate (ASR) is the share of harmful prompts where a judge says the attack worked. So every ASR is a judge’s verdict. Judges ask one of two questions: did the model refuse, or is the answer harmful? On the same answers, the two questions give different numbers. Souly et al. (2024) find that most earlier automated judges overstate jailbreak success compared with human ratings. Refusal-phrase matching overstates it most.

The Datasets Behind LLM Safety Numbers

Published:

A refusal rate or attack success rate (ASR) from one paper compares with another only when both use the same prompt set, split and judge. ASR is the share of harmful prompts where the judge says the attack worked. A split is a fixed subset of a set, such as a test split or a public release. HEx-PHI and AdvBench both hold harmful requests. They differ in categories and wording, so one model gets a different refusal rate on each. An AdvBench result also says nothing about XSTest, where the prompts are harmless.

publications

Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants

Published in USENIX Security Symposium (USENIX Security '23), 2023

A security-driven user study (N=58) measuring whether LLM code assistants like OpenAI Codex lead student programmers to write less secure low-level C code. We find the security impact is small.

Recommended citation: Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. (2023). "Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants." 32nd USENIX Security Symposium. https://www.usenix.org/system/files/sec23summer_sandoval.pdf

Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models

Published in arXiv preprint (arXiv:2509.14271), 2025

A 2022 study of prompt injection and goal hijacking attacks against GPT-3-era models, introducing Adversarial Fine-Tuning as a defense that drove attack success from 31% to near zero on smaller GPT-3 variants.

Recommended citation: Gustavo Sandoval, Denys Fenchenko, and Junyao Chen. (2025). "Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models." arXiv preprint arXiv:2509.14271. https://arxiv.org/abs/2509.14271

Surgical Repair of Insecure Code Generation in LLMs: From Mechanistic Diagnosis to Deployment-Ready Intervention

Published in arXiv preprint (arXiv:2604.16697), 2026

LLMs that write insecure code can often correctly explain the very vulnerability they just introduced — a “Format-Reliability Gap.” We trace this to a single layer and use per-vulnerability steering vectors to cut insecure generation by up to 74%.

Recommended citation: Gustavo Sandoval, Brendan Dolan-Gavitt, and Siddharth Garg. (2026). "Surgical Repair of Insecure Code Generation in LLMs: From Mechanistic Diagnosis to Deployment-Ready Intervention." arXiv preprint arXiv:2604.16697. https://arxiv.org/abs/2604.16697

radar

talks

teaching

Introduction to Operating Systems

Undergraduate course, New York University, Computer Science and Engineering, 2023

This course is an introduction to operating system design and implementation. We study operating systems because they are examples of mature and elegant solutions to a complex design problem: safely and efficiently sharing system resources and providing abstractions helpful to applications.

For the processor, memory, and disks, we discuss how the operating system allocates each resource and explore the design and implementation of related abstractions. We also explore techniques for improving performance and reliability. For example, we discuss scheduling policies for better utilizing the processor, virtual memory techniques for managing memory utilization, and caching techniques for improving disk performance.

Programming assignments using both Assembly and the C programming language will provide hands-on experience implementing core operating system components in a realistic development environment.