It'd be interesting to scale this up to a lot of images - and have users blind A/B test, like the LLM Arena on Huggingface which helped filter out "benchmaxxing" vs actual intelligence & utility. Because 1) images can be so diverse* and 2) an intrinsic tradeoff might exist between general efficiency and out-of-distribution robustness.
It's hard to rigorously judge a codec from a handful images, especially given the large impact they would have (with a widely adopted codec encoding bazillions of images in the coming years).
* There are many intricate details that go into codecs and visual perception. For instance, our sensitivity to visual errors are dependent on colors (all of hue, saturation, and brightness), spatial frequency of artifacts, local contrast and texture presence, et cetera.
And like OP mentioned, perceptual error metrics (including SSIMULACRA2) aren't perfectly representative of human-evaluated visual quality. For instance, libjxl deliberately makes choices that hurt SSIMU2 scores, but improve human-rated quality.
It's hard to rigorously judge a codec from a handful images, especially given the large impact they would have (with a widely adopted codec encoding bazillions of images in the coming years).
* There are many intricate details that go into codecs and visual perception. For instance, our sensitivity to visual errors are dependent on colors (all of hue, saturation, and brightness), spatial frequency of artifacts, local contrast and texture presence, et cetera.
And like OP mentioned, perceptual error metrics (including SSIMULACRA2) aren't perfectly representative of human-evaluated visual quality. For instance, libjxl deliberately makes choices that hurt SSIMU2 scores, but improve human-rated quality.