adeon

I played this game a bit back in 3.4.3 times.

It's been great on long-haul flights to play on the laptop. Doesn't demand your battery.

I get the vibe that it's more like there's unexpected complexity and it's difficult to be confident you know how zipcodes work with enough detail to make the feature work. And that is just one example of possible complexity.

Do zipcodes change for example? Can your drop-down quickly go out-of-date? You'd need a way to manually enter a city so people are able to tell the system an address. Do you want to bother making an auto-updating zipcode feature just for a form?

Is it going to confuse people because nobody else has bothered to make this superfancy selection feature thing?

Is this USA only? There are postal codes/zipcode-equivalents in other countries.

It starts to feel it's likely not worth the time and effort to try to be smart about this particular feature. At least not if I'm imagining this us some generic, universal address web form that is supposed to be usable for USA-sized areas.

To me it feels similar to that famous article about what you can and cannot assume about people's names; turns out they can be way more complicated and weird than one might assume.

Although maybe zipcodes don't really go that deep in complexity. But on the spot I would not dare to assume they are.

Yeah I agree. I think even if you block CSP images, attacker could still hide information, or attempted exfiltration.

The post got me now instead wondering how to not make people shallowly dismiss perfectly fine articles for dumb reasons, like I almost did. It's not even that unclear what the attack is, in the article's its opening when I look at it now again, and I now went around their posts to see how PromptArmor generally does their writing because I got curious about the writing part...

I've seen in the past vulnerabilities that were way overblown but hyped up, so this made me notice how that armor has made me be skeptical whenever some article like this feels it combines marketing + vulnerability reporting.

I found the tone in the article annoying, but my skim reading was that it is an actual vulnerability. The screenshot from OpenAI loads an image from a third-party site and the URL of the image might have all sorts of details etc.

I think the viewer should have some CSP policy in place to not do that.

That being said, if it was closed as "Not Applicable" it gives me a bit of reason to wonder if some crucial details about the whole chain was either not articulated or mentioned by PromptArmor. Maybe for other reasons it is not actually reasonable to put that on OpenAI site. I'm not sure on the spot. But on a skim read it looks like a legit vulnerability from OpenAI's part that they should fix.

I really wish PromptArmor just opened with "OpenAI's log viewer page lacks CSP policies, so it can load arbitrary URL images and here is an example how such things can easily end up on that page". This was really annoying to read but I kept going because I was curious was it a legit thing or not...

Edit: I don't know if the article was edited just now but there is a clarification paragraph that actually makes it a bit more clear. PromptArmor if you are reading this, I wonder if my gut reaction of being skeptical simply because of the tone and presentation is a common thing and there are ways to both be convincing right at the start of an article, but still allowing yourself to be marketing-like. I probably would have started with a paragraph that dryly describes exactly the vulnerability "OpenAI's Log viewer is not secure against maliciously crafted logs, which can result in data exfiltration. On this page, we show a realistic scenario by which a malicious third-party can sneak in an image URL to this page and exfiltrate data." and then go on with the rest of the article.

The task of riding a horse can be almost entirely offsourced to the professional horse riders. If they take your carriage from point A to point B, sure, you care about just getting somewhere.

Taking the article's task of essay writing: someone presumably is supposed to read them. It's not a carriage task from point A to point B anymore. If the LLM-assisted writers begin to not even understand their own work (quoting from abstract "LLM users also struggled to accurately quote their own work.") how do they know they are not putting out nonsense?

After I wrote the comment, I pondered that too (trying to think examples of what I called "security conscious design" that would be in the LLM itself). Right now and in near future, I think I would be highly skeptical even if an LLM was marketed as having such feature of being able to see "unsanitized" text and not be compromised, but I could see myself not 100% dismissing such thing.

If e.g. someone could train an LLM with a feature like that and also had some form of compelling evidence it is very resource consuming and difficult for such unsanitized text to get the LLM off-rails, that might be acceptable. I have no idea what kind of evidence would work though. Or how you would train one or how the "feature" would actually work mechanically.

Trying to use another LLM to monitor first LLM is another thought but I think the monitored LLM becomes an untrusted source if it sees untrusted source, so now the monitoring LLM cannot be trusted either. Seems that currently you just cannot trust LLMs if they are exposed at all to unsanitized text and then can autonomously do actions based on it. Your security has to depend on some non-LLM guardrails.

I'm wondering also as time goes on, agents mature and systems start saving text the LLMs have seen, if it's possible to design "dormant" attacks, some text in LLM context that no human ever reviews, that is designed to activate only at a certain time or in specific conditions, and so it won't trigger automatic checks. Basically thinking if the GitHub MCP here is the basic baby version of an LLM attack, what would the 100-million dollar targeted attack look like. Attacks only get better and all that.

No idea. The whole security thinking around AI agents seems immature at this point, heh.

Thanks for the resources. Got back to procrastinate on HN and checked the resources (briefly looked at transcript on the video, but found this article more interesting).

I've always assumed that some amount of unencrypted HTTP traffic is going to be slurped into archives, but I've been too lazy to really check an example and how does that look like in the real world. That BADASS system is an example, focusing on phones. I've also run mitmproxy in my home to learn and then I've wondered if the big agencies have something like that but much more scaled and sophisticated.

I've recently got into studying security, deobfuscated code, or decompiling, tried to find vulnerabilities or bad security, in websites and programs. I've found some, although not anything worth writing home about. I found a replay attack in one VSCode extension that implemented its own encrypted protocol, but it is difficult to use it to do real damage. Found a bad integrity check library (hopelessly naive against canonicalization attack) used by another VSCode extension. I've found something weird in Anthropic's Claude website after you log in, but because their "responsible security policy" is so draconian, I don't want to bother trying to poke it to research it further in case I earn their wrath.

Biggest bummer I found that a video game (Don't Starve Together) I had played for a long time with friends does not have any encryption whatsoever for chat messages to this day. (People gonna say private things in video game chats). The other video game I play in multiplayer a lot, Minecraft, has encryption (a bit unusual encryption but it is encryption).

That article gave me a bit of validation that I'm not a nut for giving shits about encryption and security, and being annoyed at ungodly amount of analytics I see in mitmproxy my laptop is blabbering about.

Ah yeah, I saw the propublica as well, it was one of the first articles I found when looking on the topic. I don't doubt at all that Angry Birds data was used by NSA, doesn't seem controversial.

The specific question I am interested in is: Did Rovio knowingly and willingly accept $$$ from NSA (directly or indirectly) to weaken their security? I.e. were they acting as a willing accomplice.

Because that part would be unusual for Finland (well, at least as far as I know). For US companies I wouldn't bat an eye at news like this.

Is there any reliable source for NSA paying Rovio other than this random bar discussion? Not that I don't believe you or that I'm naive about NSA and the power of money, but I looked around news in 2014 and the accusations against Rovio specifically are a bit different flavor. It seems that Rovio was oversharing data to ad networks (Millennial Media comes up a lot), and NSA likely slurped data from the advertising companies. This bar banter is suggesting that NSA had some kind of arrangement with Rovio directly instead, and Rovio willingly went along.

Or alternatively, do you feel the Rovio employee's blabbering was talking about an actual, real NSA deal with Rovio, or was it more like a bar joke and direct NSA co-operation was not really implied? (e.g. "we know our security is bad, but these ad companies pay us $XX million to not use encryption so it's sorta like NSA pays us to keep it that way sips beer").

I'm interested, because if that is an actual thing that happened, then that's an example of NSA paying a Finnish company $$$ to weaken their security, and the Finnish company willingly agreeing to that. Is it in NSA's Modus Operandi to approach and then pay foreign companies to do this sort of thing?

Your comment is describing it in few words, but to me it sounds like it maybe wasn't implying an actual NSA direct co-operation, more like someone doing bar banter and being entirely serious. But that's just me trying to guess tone.

(I'm Finnish. I want to know if Rovio has skeletons in their closet. So I can roast them.)

Okay, now that makes sense. I actually put your example of gun violence in google with kabuki theater and that found me some (depressing) articles that use the phrase. Thanks for educating me on a new expression :)