lorenzoguerra

I believe that this choice is due to two main reasons. First, it's (obviously) a marketing strategy to keep the spotlight on their own models, showing they're constantly improving and avoiding validating competitors. Second, since the community knows that static benchmarks are unreliable, it makes sense for them to outsource the comparisons to independent leaderboards, which lets them avoid accusations of cherry-picking while justifying their marketing strategy.

Ultimately, the people actually interested in the performance of these models already don't trust self-reported comparisons and wait for third-party analysis anyway

it's 100% for what they call "deep scan", it's 66.7% for the "automatic scan". Practically unusable anyway

>We also ran the full dataset of 263 URLs (254 phishing, 9 confirmed legitimate) through Muninn's automatic scan. This is the scan that runs on every page you visit without any action on your part. On its own, the automatic scan correctly identified 238 of the 254 phishing sites and only incorrectly flagged 6 legitimate pages.

...so it has a false positive rate of 67%? On a ridiculously small dataset?

I cannot bring myself to trust unreviewed software enough to install it on my own machine with arbitrary permissions. I understand the push for AI-generated websites, because the code running in my browser's sandbox is gonna have very limited permissions to do anything evil, but desktop apps are a completely different story