Risk assessment
Evals: domains, quality, elicitation
Weighted 55% of category
 | 50% | Anthropic does model evals for dangerous capabilities; some of the evals seem good, but the elicitation quality is unclear and it's generally very unclear why Anthropic thinks they help rule out dangerous capabilities more |
 | 50% | DeepMind does model evals for dangerous capabilities; the evals themselves seem fine, but it generally doesn't report enough details or explain how it interprets results more |
 | 50% | OpenAI does model evals for dangerous capabilities; the evals themselves are fine, but the elicitation quality is unclear, and OpenAI's interpretation of the results is confusing, especially for bio more |
 | 10% | Description not yet updated for Code World Model more |
 | 10% | xAI does some basic evals for chem, bio, and cyber capabilities as well as some honesty-related propensities. They're pretty bad. more |
 | 2% | Microsoft is planning to do evals some dangerous capabilities, but its plan is very vague more |
 | 0% | No, DeepSeek doesn't seem to have done evals for dangerous capabilities or have a plan |
Evals: accountability
Weighted 25% of category
| 15% | No accountability for internal evals; some external evals but details unclear more |
| 1% | No accountability more |
| 20% | No accountability for internal evals; some external evals including METR evals with good access more |
| 0% | No accountability |
| 0% | No evals or accountability plan |
| 0% | No evals or accountability plan |
| 0% | No evals or plan |
Adversarial evaluation for alignment
Weighted 10% of category
Model organisms
Weighted 10% of category
| 75% | Anthropic does model organisms research and related "alignment auditing" research more |
| 10% | DeepMind pubishes a little research related to model organisms more |
| 0% | OpenAI doesn't do anything on this |
| 0% | Meta doesn't do anything on this |
| 0% | xAI doesn't do anything on this |
| 0% | Microsoft doesn't do anything on this |
| 0% | DeepSeek doesn't do anything on this |