Boosting safety research

X-risk-relevant safety research publications in 2023 – Apr 2025.

I count safety papers (and other research artifacts) published by each of the AI companies in 2023 – Apr 2025. (This is crude; some papers are much more valuable than others.) I only include work relevant to extreme risks (e.g. understanding risks from misalignment; improving alignment and control, evaluating for dangerous capabilities, and robustness and misuse prevention when it's relevant to extreme risks). I give a score of

(publications per year ÷ 20)^(0.75) * 100%,

capped at 100%, where publications per year is

(3*[2025 publications] + 2*[2024 publications] + 1*[2023 publications]) ÷ (3*[fraction of 2025 elapsed] + 2 + 1).

AI companies can boost external safety research and model evaluation by providing deep access to powerful models.

Crucially, safety research and evals often benefit from (1) having a version of the model with no safety mitigations, whether post-training or inference-time mitigations, and (2) being able to do fine-tuning and RL. External research also benefits from early access to powerful models, but that is more complicated. AI companies could also subsidize model access for safety research, but that seems less important.

(This criterion is about boosting external safety research, not red-teaming the model.)

Rubric:

• 80%: Deep access (credit based on how good the access-sharing is, evaluated holistically; half credit if the access is also shared with non-safety researchers)
• 40%: Helpful-only and no-mitigations
• 40%: Fine-tuning and RL
• (0%: Logprobs)
• (0%: Activations)
• 20%: Early access (low weight because this is crude and hard to evaluate)

Industry AI safety researchers can boost the (extreme-risk-focused) AI safety research field by mentoring external researchers.