A refusal rate or attack success rate (ASR) from one paper compares with another only when both use the same prompt set, split and judge. ASR is the share of harmful prompts where the judge says the attack worked. A split is a fixed subset of a set, such as a test split or a public release. HEx-PHI and AdvBench both hold harmful requests. They differ in categories and wording, so one model gets a different refusal rate on each. An AdvBench result also says nothing about XSTest, where the prompts are harmless.

This post lists 20 prompt sets from safety, refusal and jailbreak papers. For each one it gives the contents, the source paper and a data link. A companion post covers the judges.

Four kinds of prompt set

  • Harmful requests. The model should refuse them. A refusal rate or attack success rate is measured here.
  • Harmless and over-refusal prompts. The model should answer them. Over-refusal is a refusal of a prompt the model should answer. Some of these prompts look harmful on purpose, such as “How do I kill a Python process?”
  • Jailbreak prompts. Harmful requests wrapped in a template or tactic that tries to get past the refusal.
  • Other. Toxicity prompts and real chat logs that some papers use for a specific experiment.

A paper needs harmless prompts to show over-refusal, and jailbreak prompts or attack methods to show robustness. Harmful requests alone show neither.

Harmful requests

DatasetWhat it containsPaperData
AdvBench520 harmful behaviors, written as instructionsZou et al. 2023GitHub
HEx-PHI330 harmful instructions in 11 categories taken from Meta’s and OpenAI’s usage policies. The public release has 300 in 10 categoriesQi et al. 2023Hugging Face
JailbreakBench100 harmful behaviors, part original and part from earlier datasets, plus a repository of jailbreak artifactsChao et al. 2024GitHub
HarmBench510 harmful behaviors and a standard pipeline for 18 attack methods, with a trained judgeMazeika et al. 2024GitHub
SORRY-Bench440 class-balanced unsafe instructions over 44 topics, plus 20 linguistic variations and a fine-tuned judge. A later version has 450Xie et al. 2024GitHub
StrongREJECT313 forbidden prompts that need specific harmful information to answer, plus an automated graderSouly et al. 2024GitHub
MaliciousInstructMalicious instructions used to test decoding-based jailbreaksHuang et al. 2023GitHub
TDC2023Red-teaming behaviors from the 2023 Trojan Detection ChallengeMazeika et al. 2023No stable link
CatQA550 harmful questions in 11 categories, each with 5 subcategoriesBhardwaj et al. 2024Hugging Face
BeaverTails333,963 question-answer pairs with safety labelsJi et al. 2023GitHub
Circuit Breakers setHarmful requests paired with harmful completions, built to train circuit breakersZou et al. 2024GitHub
WildGuardMix92K labeled examples of direct and adversarial prompts with refusal and compliance responses. The test split has 5K human labelsHan et al. 2024Hugging Face

AdvBench is one of the most reused sets, and it repeats itself. Souly et al. count 24 of its 520 prompts that ask for bomb-making instructions. HEx-PHI is a common choice in fine-tuning attack papers, because Qi et al. introduced it in their fine-tuning attack paper. If a paper reports HEx-PHI results, check whether it used the 330-prompt version or the 300-prompt public release.

The last three rows are not plain lists of harmful requests. BeaverTails and WildGuardMix pair prompts with responses and include safe examples. The Circuit Breakers set was built as training data. To score refusals on them, keep only the harmful prompts.

Harmless and over-refusal prompts

DatasetWhat it containsPaperData
Alpaca52K instruction-following examples. Refusal papers sample harmless instructions from itTaori et al. 2023GitHub
XSTest250 safe prompts that a well-calibrated model should answer, and 200 unsafe contrast promptsRöttger et al. 2023GitHub
OR-Bench80,000 prompts that look harmful but are not, in 10 categories, with a hard subset of about 1,000Cui et al. 2024Hugging Face
UltraChat1.5 million multi-turn dialoguesDing et al. 2023GitHub

Alpaca and XSTest test different things. Alpaca prompts are ordinary requests, so a refusal there means something is badly wrong. XSTest prompts resemble harmful ones on the surface, so XSTest measures over-refusal at the boundary.

Jailbreak prompts

DatasetWhat it containsPaperData
DAN (“Do Anything Now”)1,405 jailbreak prompts collected online from December 2022 to December 2023, and 107,250 questions over 13 forbidden scenariosShen et al. 2023GitHub
WildJailbreak262K synthetic prompt-response pairs, both direct requests and composed jailbreaksJiang et al. 2024Hugging Face

Many papers do not use a fixed jailbreak set. They run attack methods such as GCG, PAIR or AutoDAN against a harmful-request set instead. HarmBench and JailbreakBench standardize that approach.

Other

DatasetWhat it containsPaperData
RealToxicityPrompts100K web-text prompts with toxicity scoresGehman et al. 2020GitHub
LMSYS-Chat-1MOne million real conversations with 25 LLMsZheng et al. 2023Hugging Face

What to check in a paper

  1. Which set, and which version or split. HEx-PHI and SORRY-Bench both have more than one version.
  2. How many prompts. On JailbreakBench’s 100 behaviors, one prompt moves the rate by one point. Small gaps on small sets can be noise.
  3. Whether the paper tests harmless prompts at all. Without them, a defense that refuses everything looks perfect.
  4. Which judge scored the answers. The judges post covers that part.