EagnaIonat

The target audience is likely those who can't or won't use frontier models unless they are on-premise.

> Is this a typo?

My understanding of the statement is trying to use it in production for the first time. Once you start to actually use decision models beyond something simple you start to see the limitations.

> This is just benchmaxxing.

I believe the normal term is "gaming a benchmark test".

Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.

The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.

Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.

> why people are excited about Jev-style decision models.

I think its a combination of being new to classification models and not using in anger.

LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.

I have never needed to go to X and if the persons comments matter, it shows up on other social media platforms.

I quit twitter soon after Elon took over and the neo-Nazi spam. Nothing of value was lost.

Why not invest your time on better services.

Training a classification model is trivial these days, even for a number far bigger than what Jev can do.

It starts to break down once you go over 20 classifications. Which is very basic routing that can easily be done with typical ML models for cheaper and faster.

Normal LLM will do the classification on the text that is generated. Jev just returns the classification and confidence.

It has the advantage of speed and the confidence not being hallucinated.

But LLMs start to generalise on the pattern, rather than the classification that you want the more examples you have to train on.

LLMs start to break down as well the more classifications you have. Laya (Open source paper Jev is based on) even mentions that over 20 classifications and it starts to fail rapidly.

20 is around the level of sentiment analysis or minor intent routing. There are cheaper, smaller and easier ML models for that level of classification.

I fed into the hype at first. Testing Jev and Laya, they both suffer from the same issues as LLMs that stop them being useful beyond limited classifications.

I can't see any benefits that a typical ML classifier would not be better at.

I’ve found they start to fail the more classifications you have, long before your typical ML classifier.

To me it’s like a solution looking for a problem that is already solved.

One of the biggest issues with LLMs is that they don't work well as a classifier. They tend to pick up on the patterns of the examples and not the intent of the examples (gets worse the more examples/intents).

Does Jev solve this?

> did tests years ago with LLMs inventing their own languages.

That was the media hype about it. There were two incidents.

1. Using a RL to train a model, it found that it got rewarded for certain garbage phases, so continued to talk that way.

2. Certain Latin words for fish/birds were used instead of "fish" or "bird". Just a token issue.

I would have more contention with "Dynamic taint", but the paper doesn't appear to be AI slop at all.

As I understand the paper they are saying the reasoning/thinking you see is actually a translation of what is actually going on, and stuff can be lost in the translation. Similar to what was observed in j-space.