hobofan
Joined 12 years ago
Joined 12 years ago
If you use an odd/fringe browser you are more likely to be caught by CF Turnstile and be locked out from websites.
However you also need a few other ingredients to ensure isotropic expansion, or otherwise the expanded sample isn't scaled uniformly in all axis and you get a skewed view of the sample.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
At least that's roughly what the hypothesis of https://www.discoveryloop.com/ is.
I think the scarier alternative may be OpenAI forever gating their super-frontier models, capturing a significant portion of the sciences and leveraging it for equity in businesses that build on scientific advances. e.g. "Hey CATL, how about you guve us 5% equity in exchange for a battery chemistry breakthrough our AI has found?".
It's roughly equivalent to the playbook Palantir had for the public sector when they marketed themselves as a company that had cracked data science. At least that's what they've been pitching back then. I can't find a lot on public disclosed deals based on that.
I have some business automation process for some customers that I maintain, where Terra with low reasoning ended up with the best price/performance & latency. With an equivalent Luna with high reasoning, price was usually better, but quality had a higher variance (very much depends on where the thinking takes the model) and latency for high reasoning was worse.
Enterprise doesn't care about marginally better. They care about cost.
Most organizations that run our self-hostable chat platform with cloud models are for most stretches running 6 months-1 year old models, and are strongly eyeing (or have already switched) to more commodity models for most of their workforce. Frontier models are mainly relevant for engineering tasks.
I'm just cautioning other people to put any stakes into a language that's from its organizational bones doomed to fail for any outside party.
Putting stakes into a language that will essentially be DOA, like the author does seems like a crazy decision.
You can also just do "uv run", and it will automatically sync as necessary!
I think that's one of the things that uv does great and often goes unnoticed: It's hard to use wrong for a layman. Forgetting to run the equivalent sync command in most other Python package mangers could have resulted in a myriad of different bugs caused by out of sync packages, leading to hours of wasted debugging.
The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.
[email protected]
https://hobofan.com
Mastodon: @[email protected] (Twitter: @hobofan)
---
Currently building https://github.com/EratoLab/erato