We wrote some thoughts on static vs. dynamic evals and how it relates to understanding the security posture of an AI system. Static security evals no longer carry the signal they used to. A one-shot pass/fail tells you almost nothing about real-world risk.
Would love your thoughts on this: https://fabraix.com/blog/adversarial-cost-to-exploit