Comment by speerer

The part where the LLMs (actually, VLMs) outperformed the CNN was that the LLMs were better at avoiding false negatives (see the quote below). That kind of Type 2 error is the most dangerous one since it persuades people not to get advice. Since the LLMs are designed to be general-purpose and public-facing, the headline could be flipped to: "A specialised 10-year old CNN is a worse public facing tool than a general frontier LLM". Which would surprise nobody.

                                 | GPT |Claude | CNN |  
    Unsafe lesions kept (of 100) | 97 | 96     | 77 at its preset [sic] threshold; 98 or 99 when matched |