Well, it's good that they didn't completely forget that We Must Pace The Frontier ever existed, even though OpenAI doesn't look like it's going to join in. Some stuff I noticed:
- they still use the concerningly precise term "employee-like access", which can mean anything, rather than e.g. "employee-level". Similarly, the evaluators get tools "roughly on par" (?) with the ones actual employees use.
- they get to redact secrets from METR's reports, but METR is otherwise free to publish and whistleblow. Seems fine.
- they are going to get a separate dedicated evaluator specifically for bioweapon risks
- they still use the concerningly precise term "employee-like access", which can mean anything, rather than e.g. "employee-level". Similarly, the evaluators get tools "roughly on par" (?) with the ones actual employees use.
- they get to redact secrets from METR's reports, but METR is otherwise free to publish and whistleblow. Seems fine.
- they are going to get a separate dedicated evaluator specifically for bioweapon risks