lambda

There are a handful which restrict users to their own user accounts. But it's on dev server with friendly devs, so even for that use case it's not really the end of the world.

I'm sure our infosec folks are going to demand that we upgrade everywhere, however.

Comment · 8 days ago

Even with your explanation, this whole thing seems weird and scammy. What's the point of submitting something that is so incomplete and heavily tailored to this one benchmark outside of self promotion?

Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.

This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.

Comment · 8 days ago

This tool is for accessing Twitter/X, owned by Elon Musk, a far right billionaire.

Comment · 9 days ago

Oh, nice. I was thinking of setting up a proxy for this for testing, nice to just have it working out of the box.

Comment · 11 days ago

I think you might be in a bubble if you think that most people don't hate AI. A lot of people hate it with a passion.

https://news.gallup.com/poll/712751/americans-cool-toward.as...

39% of people think it does more harm than good, compared to 9% who think it does more good than harm. Now 52% are equivocal, but as you can see, dislike of AI is much more common than like of AI.

Comment · 11 days ago

I've even seen videos that were in English, but mislabeled, so they got auto-dubbed back to English, replacing the real voices with a monotonous synthetic voice.

I appreciate the auto translated captions sometimes as they let you get the gist of what's going on, but auto dubbing is pretty much never desirable

Comment · 11 days ago

in practice, as actually used, LLMs are nondeterministic.

It is possible, if you control the execution environment sufficiently, and are careful not to use certain kinds of parallelism which can lead to nondeterminism, and you set the seed deterministically and ensure that the prompts are byte identical, you can operate them deterministically.

But no one actually does all of that. Almost everyone uses cloud hosted APIs where the provider could change any number of variables at any time. Everyone wants faster cheaper inference so uses inference engines which may trade off a certain amount of determinism for more speed. Different versions of harnesses and chat templates and benchmark software may introduce small differences in text.

And even though the cloud providers have in the past offered seed and system fingerprints to try to give people a way to control or detect non-determinism (see https://developers.openai.com/cookbook/examples/reproducible...), that has all been deprecated and removed in the newer APIs.

So yeah, while in theory it's possible that if you control the whole stack you can get deterministic results, that's not how anyone is using it, and there is no way for third party benchmarks to ensure deterministic results of hosted models, the latest APIs don't even support it at all.

This whole "well, actually, LLMs are deterministic if you hold them right" is a red herring. In any actual real-world use, they are basically always non-deterministic.

And of course, there are reasons that providers have dropped these attempts at providing deterministic results; because the results are chaotic, perturbed by small changes in input, it's almost never useful to try to make them act deterministic or detect when non-determinism has happend. Instead, it's better to think about them non-deterministically, and look at the distribution of results, rather than just considering a single pass or fail on a single test to be meaningful.

Comment · 12 days ago

Because they selectively choose which benchmarks to highlight and which competing models to compare against; they don't always compare against the SOTA or the Pareto frontier of performance/price, a lot of times you'll see them compare against the last generation of competing models, or the like, so that they look better if you don't carefully consider what they're comparing against.

For example, here you see they are comparing against Opus 5 and GPT-5.6 Sol, when Opus 5.5, Sonnet 5.5, GPT-6 Astra, and GPT-6.1 Sol would be the latest SOTA models from those labs to compare against.

Also because a lab's own benchmarks tend to run about 6.4% better than third party runs of the same benchmark. Source: https://whatmodel.sh/claims This can happen due to a number of factors, and doesn't necessarily indicate anything nefarious. It could just be that they run multiple times while fixing issues that aren't related to the model, and have a selection bias towards stopping when the score is on the higher side. But since you see that sometimes their own scores are lower than the independent runs, some of the variation is explained just by variance in the model solving problems, since LLMs are nondeterministic there can be variation from run to run and you don't see labs always cheating, just potentially a bit of selection bias about when they stop tweaking their benchmark setup and publish the results.

And yes, there are independent third party runs of benchmarks for just this reason. The most popular with broad coverage of a range of benchmarks is https://artificialanalysis.ai/ but there are a number of others as well.

Comment · 24 days ago

This post is also dated in July, which would have been before he passed away if that post was accurate.

Comment · 25 days ago

Why would you think that the author was speaking for anyone but themselves?

When I say "juggling is fun", I'm talking about my own experience. I'm not saying that everyone else should drop their hobbies and take up juggling.

Why should the author have to couch the headline of their article with this kind of defensive language that you suggest? The very first word of the article is "I", it is clear that the author is talking about their own experience.

Comment · 1 month ago

I have not. It's something that I've considered, but never actually have done it.

Comment · 1 month ago

No, it's down again: https://xcancel.com/

They did come back for a while after the original cease and desist, but they've shut off again "due to a new development in the ongoing legal proceedings" a few days ago.

Oh, sorry, you said https://xxcancel.com/ with an extra "x".

Looks like that's a Nitter instance redirector that redirects to a few still up Nitter instances. Good to know.

Comment · 1 month ago

For those without X accounts (now that Nitter an XCancel are down): https://archive.ph/GQLvx