brianmcnulty

Long time no see! It's been a while since I've looked at MCP, but I think this does a really good job at making MCP more secure for organizations and addressing some of the weaknesses of dynamic client registration. Now that clients and approved redirect URIs can be setup directly by the IdP and organization, a lot of the attacks that were possible with DCR (confused deputy, phishing attacks, etc.) can be mitigated more broadly. It also makes it so servers don't have to implement as much authorization logic as they did before if the IdP or organization didn't support DCR, which is a pretty big advantage (especially if they combine MCP auth with existing API auth).

One major downside is consumer usage seems to still need DCR with this. I think this could potentially be addressed by existing consumer OAuth providers (Sign in with GitHub, GitLab, Google, etc.) adding support for registering static MCP clients/servers, clients shipping their static client IDs inside them, clients allowing users to sign in with GitHub/GitLab/whatever IdP, and letting the user self-manage connections on the IdP's site.

Overall, XAA/EMA seems vastly superior to DCR from a security perspective (and also usability too, since users don't have to configure as much!). The concerns I have are also much easier to address and have way less security impact than with DCR, since attackers don't get to register their own clients anymore and there are less pitfalls for MCP server developers.

Why would you have ethics when you could get that IPO money instead?

I wonder how Claude Fable will live up to expectations and how good those Fable/Mythos classifiers really are. It seems a bit convenient for Anthropic to release this magical insane model when they are about to IPO.

They do this by allowing you to download all of the components (minus data cryptexes containing the model weights) and run it on your own Apple silicon chip (you can put your computer in recovery mode and use csrutil to enable research guest operating systems)

I think what is concerning is that they are expanding into Google Cloud and NVIDIA to run with it too with their versions of confidential compute, which if I remember correctly are not as well verified as Apple PCC and a little harder for researchers to get their hands on.

Apple uses a key ceremony process where no single party has access to all the keys required to sign hardware, meaning in theory they can’t just sign malicious hardware. However, I’m not sure how Google and NVIDIA play into this and I don’t think they’ve provided much detail on it. I think it seems a little rushed to get the features out since they fucked up with initial Apple Intelligence release.

No, it's more that those apps needs to be able to make all of the tool calls Siri AI can make, which would allow third-party developers to collect data they shouldn't have access to.

App developers can already access the on-device foundational models through an API, but I don't think many developers want to do that because there are better models.

I think it's because Apple would have to provide every competitor (including ones running off-device with no confidential compute) with the same level of access Siri AI would get, which poses a lot of security and privacy concerns Apple would never allow third-party developers to get access to even with a TCC consent prompt (like reading and sending iMessages).

Yeah, it seems another ATO bug has popped up. I haven’t looked too much at it personally, but I hope Meta plans on taking their Meta AI Support Assistant offline until it undergoes far more rigorous security review.

Not exactly, I think part of the tools it has access to allows it to perform an "investigation" into recent malicious activity and account changes that may have occurred that were likely unauthorized (such as changing from an email used for a long period of time to a new one). I think this AI-version of this check was originally broken and just allowed any email but has now been fixed to only be emails that look like they were attached to the account, which is what the poster used to obtain access back into their account.

You should also assume the user can read any data you send back from a tool call or data you add to a user response. If any part of the input or output is controllable by an attacker, you should be assuming some prompt injection is possible that allows them to access all data and tool calls the agent had and has access to.

I do a lot of bug bounty research on Meta and Instagram, and some of the bugs I find look extremely simple like this but have some slightly complicated reason for why they occur. Maybe not this one, but I do have a guess as to what might have actually happened.

Based on what I've seen so far, Meta AI Support Assistant (they call it "MAISA") had tool calls that a) start an email verification to any specific email, phone number, or the contact points linked to an account and b) allow generating a password reset link for an account based on an email verification attempt. I don't think it had any access to the actual codes themselves, but rather think a handle or ID for an email verification attempt (along with the user provided verification code based on user input) was provided to the "generate reset password link" tool call, and the tool call failed to properly validate the actual email used in that attempt belonged to the account allowing the ATO.

The tool call for MAISA to generate a password reset link should have failed with an email verification attempt that corresponds to an email not linked to the account (and I believe I even tested this at one point on Facebook and encountered an error that successfully prevented it), but I suspect they tried making a change to this tool call for Instagram where slightly older, recently unlinked emails could be used to recover an account that got hijacked by an attacker, which added the need to allow emails not currently linked to the account to be used and set to the user's primary email.

I also suspect that the MAISA tool call change called a wrong API or something that unintentionally allowed any email verification attempt that was successful to be used, but the engineers did not add a sufficiently thorough e2e test case to test the tool call against unrelated email verification attempts being provided to the tool call. This is the part I think should be focused on the most. Tool calls for agents that have their output potentially influenced by an attacker should be treated like external APIs that anyone can reach, and they should be tested as such.

This is all obviously a guess, doesn't take into account the many signals they use to determine if an account recovery attempt is valid, and could be very inaccurate, but it's the closest to what I (someone who deals with Meta security a lot) think could have allowed this to happen.