tehryanx

it's untarring plaintext skills. Skills are basically executable in a claude workspace.

never flagged a comment faster, gross.

Pedant here. Using "for instance" here implies that you will be providing an example of how begging the question and raising the question are not the same thing. You've provided a definition and a source, not an example.

It makes a lot of sense to use an MCP for git and everything else if you want observability across many users. It gives you a place to shim security controls, monitoring, and alerting into the tool call pipeline.

I find it even more fascinating that a few of them aren't.

newer models have larger context windows, and more stable reasoning across larger context windows.

If you point your model directly at the thing you want it to assess, and it doesn't have to gather any additional context you're not really testing those things at all.

Say you point kimi and opus at some code and give them an agentic looping harness with code review tools. They're going to start digging into the code gathering context by mapping out references and following leads.

If the bug is really shallow, the model is going to get everything it needs to find it right away, neither of them will have any advantage.

If the bug is deeper, requires a lot more code context, Opus is going to be able to hold onto a lot more information, and it's going to be a lot better at reasoning across all that information. That's a test that would actually compare the models directly.

Mythos is just a bigger model with a larger context window and, presumably, better prioritization and stronger attention mechanisms.

I get what you're saying, but I think this is still missing something pretty critical.

The smaller models can recognize the bug when they're looking right at it, that seems to be verified. And with AISLE's approach you can iteratively feed the models one segment at a time cheaply. But if a bug spans multiple segments, the small model doesn't have the breadth of context to understand those segments in composite.

The advantage of the larger model is that it can retain more context and potentially find bugs that require more code context than one segment at a time.

That said, the bugs showcased in the mythos paper all seemed to be shallow bugs that start and end in a single input segment, which is why AISLE was able to find them. But having more context in the window theoretically puts less shallow bugs within range for the model.

I think the point they are making, that the model doesn't matter as much as the harness, stands for shallow bugs but not for vulnerability discovery in general.

I first mirrored these in the early 2000s because I was worried it would eventually vanish. my mirror has been gone for decades, and the original survives. :)

where is anthropic hyping like that? Most of what I see coming out of anthropic is deep context releases on research they're doing.

The real problem here is that this is now the only way the maintainer/reporter can reasonably work.

Proving out a security vulnerability from beginning to end is often very difficult for someone who isn't a domain expert or hasn't seen the code. Many times I've been reasonably confident that an issue was exploitable but unable to prove it, and a 10s interaction with the maintainer was enough to uncover something serious.

Exhausting these report channels is making this unfeasible. But the number of issues that will go undetected, that would have been detected with minimal collaboration between the reporter and the maintainer, is going to be high.

Rolling your own browser is 10x more dangerous than rolling your own auth or crypto. Building on top of chromium is a good thing here.