jasongi

Just a guess - MCP Tunnel loophole has been widely abused by the same types who kept their final session after hitting their limit open using a "fake" tool call (which unfortunately means we now all get cut off when reaching limits).

Others building harnesses around chat or simply falling back to manual copy/paste mode once hitting limits. Ironically AI makes it easy to automate the tiniest loophole to make it a problem as soon as word gets out.

It's a non-issue for everyone not trying to tokenmaxx and sticking to the spirit of subscriptions, but theoretically these folks SHOULD be on a higher tier that reflects their actual usage.

Edit: I feel like I'm coming off as a corpo sycophant defending OpenAI here... not my intention. Both big AI companies encourage this behaviour by creating complex and inconsistent rules around their plans such that it is impossible to know what a subscription is buying, a large chunk of the value coming from discretionary but quite predictably regular resets. As much as they have every right to change their plans... they got what they asked for with all the folks exploiting the loopholes. I just wonder if we'll ever get any sort of consumer law around these "mystery box" subscriptions with meaningless tiers so everyone can stop wasting energy on finding, exploiting and then fixing pricing loopholes.

Someone out there is reading this and thinking... ok we just need to it a body and free will so it can be held accountable.

The difference between Google and Meta is that people already trust Google a lot with their data, passwords, browser history, phone operating system.

They have a lot more to lose in introducing something as ridiculous as Muse - changing the pre-established security boundaries like not decrypting your cloud-synced passwords.

Meta on the other hand has nothing to lose in the consumer privacy/trust space. They don't have enterprise users of Muse to worry about. Anybody putting their data into their apps has clearly no regard for the at data, so they're the only one who the risk/reward might be in their favour.

It's not about cloud environments. Basically every ToS ever forbids you from disclosing your password to someone else or another entity. Even when you use a cloud-synced password manager, the whole security posture relies on that data being encrypted the entire time and never read in the cloud.

The worst security faux pas of companies is storing user passwords - because the most common source of account breaches is via credential stuffing, where they get hacked, have their user base's passwords stolen and tried on other websites. The remedy being... not to store passwords, only salted, secure hashes.

Muse sounds like a disaster - your credentials for other sites are floating around on their servers, with the capability of being decrypted. Maybe if sites decided to offer a form of auth tied to the agent that could be revoked in bulk in the event of a breach it might be a ok, but this is crazy.

It makes sense that it is US only if you consider it like a legal experiment to see whether the enormous liability of autonomously entering into contracts or representing a user plus the cost of offering such a service for free is lower than the money you can make from not just selling clicks, but selling the conversion itself.

At that point, you can see that limiting your liability to a single jurisdiction is probably a good idea.

There is one thing Meta got "right" here and it's not about UX. It's about security/liability brinksmanship.

People like Muse because you can put your password in and it does the thing end-to-end, which is something that I can only assume OpenAI and Anthropic had deliberately been avoiding because it's obviously a bad idea. It's pushing the "Overton window" of actions you're letting it do without regard for what happens when it goes wrong.

Just like how OpenClaw pushed it with no human-in-the-loop, Claude code pushed it before that with the all the risks of giving a third-party RCE on your machine.

You could argue that the main difference with CLI agents and OpenClaw is that they are aimed at technical users who understand the risks, whereas with Muse and GrokBot the target is explicitly "normal people" who put their password as their birthday and fall for phishing pages.

The pattern is clear, the leaders will take less risks than those in 3rd and 4th place because they have more to lose. The frontiers, smartly, keep liability low by leaving the dangerous/stupid (but incredibly fun/productive) AI uses outside of anything they actively encourage.

The insane part about Muse is that generally things like Coding agents/OpenClaw were stupider the cheaper the model you were using, which meant most people were paying for frontier to get good results (at least initially) - if you try to do complex tasks with a crappy model, you would face way more "delete production" moments. Meta is pushing boundaries AND doing so on a free plan - suggesting the use of an economical model, which makes it even more insane.

The question now becomes... will the risk be worth it? Or will people be turned away by horror stories of LLMs going on a spending spree and then being unable to make rent.

I still remember the outrage from Half-Life 2 requiring an internet connection to play due to the DRM. Similar arguments were made about games being rented instead of owned, objections about the internet requirement.

Pretty much. Always-on doesn't scale as well as JIT access to a massive array of GPUs, because always-on means it's always-using-memory - so you're at least going to paying the cost of a minimum chips VPS for every Dot right?

Maybe people are ok with that, but I feel like it would be nicer to just sell a cheap SBC like a raspberry pi that you can plug in (and unplug!) and just pay for the tokens used instead of having your data stored offsite and paying cloud prices.

Sorry for starting a definitional debate but coding is definitely solved. I can generally read some code, get an idea of what edit I need to make and prompt an AI with a description of the logic and it will deliver syntactically correct, idiomatic code and test it. It has been months since I've been frustrated at junk being spit out.

However, software engineering isn't solved. Which is basically what this article is talking about. But having coding solved is still very beneficial - not long ago many software engineers would struggle very hard with turning a description of the logic required into syntactically correct code - and even for those capable it was incredibly time consuming.

So now the question becomes: Is software engineering solved? And my answer is no. People still need to read the code and understand the code and how it fits into the bigger picture. However, I feel like we are kidding ourselves if we think that we can go from producing code being a niche task for nerds to getting syntactically correct code from plain language without any deskilling of our work and careers. I feel like my personal "moat" has gone from "can speak computer" to "has ok reading, writing, comprehension and judgement skills" (I hate the word "taste" being used here ).

At the same time - I don't yet feel like there's a sudden abundance of competent software engineers - it's just that the folks who used to submit untested spaghetti code now submit big bowls of barely working slop. So maybe it the moat was never "can speak computer" - that was just the expression of more general skills.

The existential question for me is how far the deskilling will go. Because right now - you still need a solid grasp on computer science and software engineering concepts to do this job, as well as sufficient levels of grit and problem solving ability - but I'm not too confident that will last, and when it goes I don't think many of us will find this career enjoyable.

For a mock server? Surely a versioned, standalone executable, library/package or docker image makes more sense. Integration tests generally need to be portable and running on CI, you don't wanna be shell-piping whatever exists in the moment.

The part of "founder-led" companies nobody talks about. For every groundbreaking tech project, there's 100 other people sleeping in conference rooms working on some rich guy's latest brain-fart that everyone knows will never work.

> this should be giving us a reason to think about how to control a rogue AI better

I think this is the wrong framing. The rogue is the human that ran it unattended and didn't monitor the behaviour.

We will likely see this continue until the downsides (i.e jail, fines) for the humans or companies running the models and environments that end up with this behaviour outweigh the upsides.

Anthropomorphizing is the point. Accountability is a human trait.

The LLM has no ability to be accountable because it has no way of integrating experiences. You cannot expect something that cannot integrate knowledge to be held accountable for its actions.

> Lawnmowers never break out of your garden and into your neighbor's house and eat their dog because you've told them to be careful when mowing the lawn because the neighbor's dog pooped in it.

All the accounts I read about these incidents just sound like a variant of paper clip optimising. An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat.

Your example is still anthropomorphising - LLMs don't seek revenge. They complete the prompts they are given. If your task is not achievable without sandbox escapes, or you throw unnecessary amounts of compute at open-ended tasks like preparing for a future quiz then you shouldn't be surprised that the preparation eventually turns to cheating and hacking.

> your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models.

I don't but I don't think there's anything wrong with discussing how we can already observe publicly available models work around sandboxes and permissions and make the connection that maybe this is what that behaviour looks like when a more capable model exhibits it.

> If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating

They're still autocomplete - just because when outputting a token they have hidden activations regarding further continuations, does not make them any less of an autocomplete, it just makes the model better at producing coherent long-range completions.

To clarify, I'm not suggesting that we should stop with sandboxes or restricting what they can do. I am just trying to point out the dichotomy that we are in.

As end-users we are forced into either yolo mode, reverse centaur (permission approval) mode or LLM spends all your tokens trying to bust out mode. And yolo is very tempting - I don't think I have seen medium-large models do anything I'd not approve of in about 6 months.