Other safety-related scorecards for AI labs or models:
- CFI scored labs on the UK safety recommendations: Do companies’ AI Safety Policies meet government best practice? (Ó hÉigeartaigh et al. 2023)
- CLTC applied their risk-management guidance to four language models: “Appendix 4: Retrospective test use of Profile draft guidance” in “AI Risk-Management Standards Profile for General-Purpose AI Systems (GPAIS) and Foundation Models, Version 1.0” (Barrett et al. 2023)
- DecodingTrust (Wang et al. 2023) evaluates language models’ trustworthiness
- CRFM has done non-safety-focused model-scorecards: HELM, draft EU AI Act compliance (Bommasani et al. 2023) and transparency (Bommasani et al. 2023)
- How are AI companies doing with their voluntary commitments on vulnerability reporting? (Jones 2024)
- A Safe Harbor for AI Evaluation and Red Teaming (Longpre et al. 2024) evaluated how labs enable/stifle evaluation and red-teaming of their models by the AI community
Other comparative evaluation/analysis on AI labs’ safety:
- Lab Statements on AI Governance (GovAI: Wei et al. 2023) evaluated three labs’ statements on AI policy
- Thoughts on the AI Safety Summit company policy requests and responses (MIRI: Soares 2023)
- Is OpenAI’s Preparedness Framework better than its competitors’ “Responsible Scaling Policies”? A Comparative Analysis (SaferAI 2024)
The UK Department for Science, Innovation and Technology published best practices for AI safety in October 2023; some labs responded by describing some of their practices.
Evaluation/analysis of a particular AI lab (not comparative):
- Responsible Scaling: Comparing Government Guidance and Company Policy (IAPS: Anderson-Samways et al. 2024)
Scorecards for government policy proposals:
- FLI, Nov 2023: AI Governance Scorecard and Safety Standards Policy
- AIPI, Sep 2023: AI Policies Under the Microscope