Other safety-related scorecards for AI companies or models:
- FLI scored labs on safety: FLI AI Safety Index 2024
- SaferAI scored labs’ risk management practices: Are AI Companies Preparing for a Safe Future? (Campos et al. 2024)
- CFI scored labs on the UK safety recommendations: Do companies’ AI Safety Policies meet government best practice? (Ó hÉigeartaigh et al. 2023)
- CLTC applied their risk-management guidance to four language models: “Appendix 4: Retrospective test use of Profile draft guidance” in “AI Risk-Management Standards Profile for General-Purpose AI Systems (GPAIS) and Foundation Models, Version 1.0” (Barrett et al. 2023)
- Scale has an adversarial robustness leaderboard
- TrustLLM (Huang et al. 2024) evaluates language models’ trustworthiness
- DecodingTrust (Wang et al. 2023) evaluates language models’ trustworthiness
- CRFM has made (non-existential) “safety” model scorecards: HELM, especially HELM Safety (blogpost) and AIR-Bench; draft EU AI Act compliance (Bommasani et al. 2023); and transparency (Bommasani et al. 2023)
- How are AI companies doing with their voluntary commitments on vulnerability reporting? (Jones 2024)
- A Safe Harbor for AI Evaluation and Red Teaming (Longpre et al. 2024) evaluated how labs enable/stifle evaluation and red-teaming of their models by the AI community
Other comparative evaluation/analysis on AI labs’ safety:
- Lab Statements on AI Governance (GovAI: Wei et al. 2023) evaluated three labs’ statements on AI policy
- Thoughts on the AI Safety Summit company policy requests and responses (MIRI: Soares 2023)
- Is OpenAI’s Preparedness Framework better than its competitors’ “Responsible Scaling Policies”? A Comparative Analysis (SaferAI 2024)
The UK Department for Science, Innovation and Technology published best practices for AI safety in October 2023; some labs responded by describing some of their practices.
Evaluation/analysis of a particular AI lab (not comparative):
- Responsible Scaling: Comparing Government Guidance and Company Policy (IAPS: Anderson-Samways et al. 2024)
Scorecards for government policy proposals:
- FLI, Nov 2023: AI Governance Scorecard and Safety Standards Policy
- AIPI, Sep 2023: AI Policies Under the Microscope
Scorecards for governments:
- Stephen Casper (Nov 2024)