crazylogger

Selling tokens might not be the best business ever, but since starting random startups is an even worse business, you'd stick with selling tokens!

The backdrop being deepseek offering 1% (I remember it was ~1% when 4-pro first came out early this year - 4-pro is now removed) / 2% (current for 4.1-flash).

Let's say Claude Code's system prompt is updated to recommend responding in Simplified Technical English. It'll work for the current generation of models. But Anthropic will train the next generation via RL on Claude Code traces as they do today. As long as they don't change their reward design, the next generation is going to be pulled towards Claudish again, because speaking Claudish gives higher rewards, so in the end the prompt doesn't really matter.

Ideally these things shouldn't work like this but this seems to be the sorry state of RL right now.

Assuming there is an index, agent still needs to write queries against the index to get any information from it - might as well grep!

Anthropic server decrypts it as part of fulfilling every request, and haiku recites it per your request.

It's not about discrediting AI. We know LLM is a commodity technology like electricity at this point. If somebody in 1900 claimed they had a setup at home where they feed in electricity and cool air comes out the other end (meaning they invented AC), obviously people would want to know what the setup is, so everybody can have AC.

They should just be symlinks to CONTRIBUTING.md and other docs humans are meant to read.

The whole idea behind LLM is they use the same language and tools as us, so we literally don't design anything specifically for them.

The goal is to test for AGI where G stands for general, that means ability to act in any environments, ideally solving novel tasks using novel tools we’ve never seen before in the world. If a specific prompt or tool design lifts a model’s score it’s a sign the model is overfitting to a particular modus operandi, therefore not general.

I think in this age where models are heavily RL-ed on acting in specific harnesses, this type of benchmark is more important than ever, to make sure they’re not in fact moving further away from general intelligence.

Certainly sounds like a "P vs NP" style conjecture that shouldn't be possible in practice, save for certain generalizations, such as "this is a cybersecurity task, we know Fable will refuse (and score zero), so we just route to K3".

> Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).

Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.

The Y axis is total cost in USD. For it to go down would mean OpenAI refunding you money.

Manufacturing is primarily in China - that's true for Go Pro & everyone else and almost needless to say. The point is China usually eats the design layer too, making Apple a little unique in that they survived Chinese competition completely unscathed.

Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.