tzone

Lol, the title is so misleading. As more information is coming out, if the Cursor Claude extension really is the culprit here, these attack must have been in the works for god knows how long.

This level of targeting is really only something that nation state actors can pull off, who are also protected from actual law enforcement prosecution. Unless attackers are really in North Korea, they would 100% be caught within just few days.

You can say that this isn't compromise of Anthropic, but it sort of is in the setup. Cursor Claude extension is marked as something made by Anthropic, so default assumption is that only people who can release that new extension would be a team at Anthropic.

Since that is clearly not the case, that is just as much Anthropic's problem as it is Cursor's problem. There either needs to be a technical change so that extensions are signed with keys that Anthropic controls, or you should take the branding off from the Cursor extensions.

Because they are using 100x more tokens with the latest best models only.

If you cranked up turns and only used fable level model for every interaction, you would get way way less jank. Plus when you have essentially unlimited tokens, you can always do extra AI review passes to reduce jank.

I think for us, outside of Anthropic, it is hard to imagine the scale of AI agents that they are running internally . We are talking about probably 10s of millions $ worth of spend monthly if not more.

There are exchanges, custodians, etc that hold insane amounts of BTC or ETH in a single address.

We are talking billion dollars+ worth in a single address. So if either chainskey security is compromised in a way that it is conceivable to brute force it, there will definitely be plenty of targets.

Well we don't know what the margins on AI models will be. If the proprietary models hit a ceiling pretty soon, then open source models and harnesses could close the gap, at that point it becomes much more of a lower margin business. And OpenAI and Anthropic will no longer be able to dominate as much as they are today. That would 100% affect their projected multi trillion valuations negatively.

It all depends on what prompt you use though. You can just tell all current frontier models to write a chess engine first, and then play a game of chess against you using that engine. It will probably do a pretty good job if you ask it that way (it will also burn a shit ton of tokens, but hey, that is part of the fun).

On that note, I actually had an overall harness (for experimenting) that was essentially like this: "for any task, instead of answering question directly, write a program to answer the question instead. test and verify the program before giving the answer".

It actually worked incredibly well on all "gotcha" LLM questions like math or counting letters in words and all sorts of stuff.

Of course it was ridiculously slow and very expensive but it was a proof of concept that it can actually be much more accurate on every task if you are willing to spend an infinite amount of money.

It all depends on what prompt you use though. You can just tell all current frontier models to write a chess engine first, and then play a game of chess against you using that engine.

It will probably do a pretty good job if you ask it that way (it will also burn a shit ton of tokens, but hey, that is part of the fun).

On that note, I actually had an overall harness (for experimenting) that was essentially like this: "for any task, instead of answering question directly, write a program to answer the question instead. test and verify the program before giving the answer".

It actually worked incredibly well on all "gotcha" LLM questions like math or counting letters in words and all sorts of stuff.

Of course it was ridiculously slow and very expensive but it was a proof of concept that it can actually be much more accurate on every task if you are willing to spend infinite amount of money.

This is one part where it should really just be done by AI for 99% of the cases. Just have a security focused AI model that is biased towards flagging things a bit more conservatively.

Only apps that get flagged by the AI reviewer then have to go through a separate, slower human review process.

With latest AI models, we will most likely see shift back to just fully Native apps even for smaller teams. This is type of stuff that AI models do really well, if you get a particular feature or even a bigger app change done in one platform, you can just tell Opus or Fable to replicate to the other platforms and in almost all cases it will just do it as well as most human teams would have done it.

Physics alone is more than enough to “further human understanding”. All current mathematicians can move to other sciences, closest being fields in physics, and it will all continue to progress just fine.

Well, if in future we do end up with a magical tool that can solve any formal mathematical problem on a whim, we really won’t need field of mathematics anymore as it is today.

There would be no need to deliver new mathematical insights by solving problems. You would just have a magical math problem solving machine and that’s it.

Clearly The goal was to scoop Anthropic not a single researcher. OpenAI heard the rumor that Anthropic solved an open problem. So they went nuts pulling all plugs to scoop them.

Turns out it wasn’t actually Anthropic and just a researcher with a single Anthropic guy friend working on it .

Wild times

While AI companies have almost infinite money, they still don’t want to blow million dollar budgets on problems if there isn’t high likelihood that it will be successful.

But within next 10 years as costs drop significantly and even more improvements are made, yes it is very likely that almost every single existing math problem will get a serious AI cracking done on it

You are glossing over the fact that there is reasonable suspicion that OpenAI used privileged information to do “the scoop”. I.e. they used the fact that researcher used OpenAI tools to get advantage .

Imagine if OpenAI opened up a high frequency trading arm and suddenly stole all the prompts and research that other HfT firms are doing through OpenAI tools and start making bank based on that . Wouldn’t that be straight up insane?