hbrn

If AI does something good, you take responsibility. If it does something bad - "the agent messed up". /s

I wouldn't be surprised if a good portion of these PRs are multiple agents stuck in a loop constantly overwriting each other changes, with no human involvement at all.

Previously there were two problems with NIH solutions:

- they have implementation cost.

- they are usually born from arrogance mixed with lack of understanding of existing solutions and their tradeoffs, and typically end up being a net negative.

Before LLMs, the cost is what primarily used to stop us from building them, it used to be a natural filter for bad ideas. Now that the cost is gone, it's nearly impossible for most teams to stay disciplined and not be building those.

Why wouldn't you build a software factory? If you succeed, the ROI is infinite. Who cares about immediate customer needs, we're building a utopia here. Once it's done, all customer needs will be solved. Just one more month, I promise.

> decisions made by SaaS providers that don't work for you

Unfortunately this starts to break when there's more than one you on the team.

I see a lot of hand-rolled NIH solutions that create value for one person only, and are a net negative on the team.

> does this advice correspond to our policy docs

This is not a “system one” question.

You’d be much better off using a proper LLM here: more accurate, can provide justification, can be steered when mistakes are made. Jev is going to be a coin flipper with almost no knobs.

> give it a list of probability for each number and ask it whats the probability of each number

Did i hear that correctly? In order for Jev to be accurate you have to give it the answer before asking for the answer?

(btw this is exactly how Jev is playing games).

Ha, two years is too generous.

If they weren't lying about $100M ARR, they had to be at 6.5T/day for a week to hit that number.

They also claimed they were at 1T/day just couple weeks ago. That's $15M ARR.

85% drop in just a couple of weeks?

These are very hard problems to solve.

It's often nearly impossible to know in advance how hard the task will be, so the model has to guess, and will inevitably miss. Just because you know how hard the task is, doesn't mean it's obvious. Often it's only obvious to you, because you have additional context that models are missing.

A bad guess can lead to either context rot (if subagent wasn't used), or slower execution (if subagent was used but it turned out to be a bad idea). Both are bad UX.

Now, we could improve accuracy by having the harness do a research on each task before executing... which leads to even slower execution, also bad UX.

At this point there's no need to pay a data scientist. You can literally ask Claude to do everything for you: extract real examples, classify them, post-train a model, and ship an API.

Now, it could be viable if your business has literally hundreds of problems thats require classification. I just haven't seen those.

I treat the fact that almost noone was doing that as evidence that decision models aren't that useful/groundbreaking. That, and the fact that every single demo I saw was either fake (e.g. playing games), contrived, or plain wrong (e.g. using Jev for compaction).

Measuring ARR based on 7 days of data is beyond stupid.

You can create a company with 2B shares and sell one share to your friend for $1000. Lo and behold, you own $2 trillion dollar company, leaving Elon behind.

Jev API returns both confidence and probabilities.

But confidence value is just a function applied to probabilities. It is not coming from the model, and it carries no additional information.

It is documented btw, and yet you will see plenty of claims that Jev is better than LLM because it returns both.

> work on one specific problem, right? That is the opposite of "general purpose".

A business doesn't need Jev for the sake of Jev. Most business are solving specific problems.

And fine-tuning got a lot cheaper these days - I've seen claims here on HN that ~500 examples is enough to beat Jev.

> "nobody thought it's a good tradeoff", which again is false, there was huge demand for a cheap and accurate general purpose classifier

There wasn't. The hope is that there was a latent demand, but we've yet to see if it's truly latent or just manufactured.

Noone is saying "hell yeah, finally we got a general purpose classifier, my business needed it so much". The typical message is "this seems cool, let me see where I can apply it".

The fact that name itself is a play on Jevons Paradox illustrates that there was no demand until Jev was released.

Yeah, but I wouldn't be surprised OpenAI's decision API is a post-trained Luna with confidence calibration.

Typesafe claims that Jev is calibrated, but there are plenty of examples where it completely fails (predicting die roll being the most obvious one).

Unfortunately calibration is hard to benchmark.

> people that came up with a product people wanted

We've yet to see whether this is true, or is it just manufactured demand. There are dozens of Jev demos, but pretty much all of them are either cool but useless, or simply fake (i.e. harness doing 99% of the work).

> Models we had before Jev had at most 2/3 of above qualities, but none of them were 3/3.

You're the one being deceptive here. Jev is trading accuracy, speed, and cost for generality. It's less accurate, slower and more expensive than trained classifiers. So it's still 2 out of 3, but with decimals. Maybe 2.2 out of 3 if I'm being charitable.

And the reason we didn't have that before is because nobody thought it's a good tradeoff.