convexfunction

If you know little about prompt engineering and want to throw together a demo of something that kind of works extremely quickly, or experiment with an LLM agent exactly as it's defined in some paper, LangChain is pretty useful.

If you want to develop a real LLM application, you're probably better off skipping the library completely, or at least fully understand each abstraction to make sure it does everything you want before you decide you want to incorporate it.

Love it. Execute the scam, immediately send the money back (or better, somehow don't accept it in the first place) and tell them what happened with a link to an informative webpage. Might even be self-sustaining if you solicit donations from the victims concurrently with revealing the fake scam (though, uh, perhaps they'd be unusually wary about giving any amount of money right then and there to some organization they'd never heard of before).

ChatGPT-4 throws some cold water on the idea, naturally, though I still think there's something there:

====

me:

i have an idea for a non-profit foundation that would leverage LLMs to perform "online scam pentesting" at scale, as a public service. basically, we would execute pretend online scams very similar to real online scams that people are victimized by (fake invoices, romance scams, etc), but immediately send the money back if the pseudo-victim falls for it along with a brief explanation of what happened and our foundation's mission, along with a link to an informative webpage with helpful resources on online scams. (if the victim ignores the pseduo-scam, we'd perhaps wait a day and then followup with the same info.)

this seems to me like it'd be potentially legally fraught, though i'm not sure exactly how, or if there's any relevant legal precedent on the topic. what do you think?

ChatGPT:

I am not a lawyer, but I can provide some general thoughts on the potential legal challenges associated with your idea. It's important to consult with a legal professional for advice tailored to your specific situation and jurisdiction.

Consent: Executing pretend online scams without the consent of the individuals involved could potentially be a legal issue. Consent might be difficult to obtain without revealing the purpose of the exercise, which could defeat the purpose of the "pentesting."

Privacy: Collecting, storing, and processing personal information might raise privacy concerns, especially if the information is sensitive. Different jurisdictions have different regulations, such as GDPR in the European Union and CCPA in California, which govern the collection and processing of personal data.

Fraud: Even though your intention is to educate and protect people from scams, the act of sending fake invoices or pretending to be someone else could potentially be considered fraudulent behavior. This could open up your organization to legal liabilities.

Unintended consequences: If someone experiences financial, emotional, or other harm as a result of your "pentesting," they might choose to take legal action against your organization.

Regulation of LLMs: Depending on the jurisdiction, there might be specific regulations governing the use of LLMs, especially for activities that can be perceived as deceptive.

Precedent: While there may not be direct legal precedents for this type of activity, similar cases involving penetration testing and ethical hacking might provide some guidance. However, those cases typically involve consent from the targeted organization, which is a key difference from your idea.

To minimize potential legal risks, you could consider alternative approaches to educating people about online scams. This could include creating awareness campaigns, providing educational resources, or partnering with organizations that share your mission to fight online scams. Regardless of the approach you choose, it's essential to consult with legal professionals to ensure compliance with relevant laws and regulations.

My mother got completely wiped out by a romance scammer, tens of thousands of dollars over >1 year, all via cash dumped into a Bitcoin ATM I believe. AFAIK she's still talking to the "guy", or she was when I broke off contact with her, after I and just about every other loved one, her bank, and the actual **ing FBI have all patiently explained many times the nature of the scam and exactly how we can tell it's a scam (beyond just "you've never met this dude or even had a video call and he's weirdly financially needy for someone who makes so much money") and sent her all sorts of articles and online resources on the topic and she reassured everyone that she recognized she'd been such a fool and would cut off all contact. And no, she wasn't displaying any other symptoms of dementia or Alzheimer's or anything like that that I could recognize.

I don't know if OpenAI/etc keep close enough tabs on usage that they'd be able to identify if they have people abusing their product for these types of scams, but even if they are I'm sure it's just a matter of time before the leaked LLaMA weights or similar are applied for this purpose, if it's not happening already.

Not a clue what's to be done collectively if law enforcement doesn't care beyond the FBI keeping an eye out for "more serious" money laundering stuff (I suspect that's why they actually care as an institution), but I can share a little practical advice that maybe you won't hear elsewhere:

1. If you're substantially financially supporting your parents or whoever else, you're entitled to and need to understand their budget in extreme detail, just like if you were married. They should be able to explain and prove all of their income and all of their expenses, and they should understand that your generosity is not entirely unconditional -- in the same way that you wouldn't be willing to enable, say, a heroin addiction, you also aren't willing to enable literal scams. (And, of course, they should understand why you need this info and what risk you're trying to protect them and yourself against.) This still doesn't protect them fully, since they can e.g. be persuaded to sell their home and send it all off in secret and only tell you about it once it's too late, or just lie and continue sending off small amounts of money over time. But, it helps limit your exposure, and if you're lucky will make them think twice about people they've never met in person soliciting money from them online.

2. If you see a loved one fall for any online scam, that should probably increase your estimation of their ongoing susceptibility to similar scams, even if they seem to "get it" after the fact in that specific case, even if you don't think of them as particularly gullible in general. Perhaps they'll learn and be more careful, or perhaps they won't and you're just learning new information about them that's unlikely to change.

3. Don't underestimate what even simulated love can make people do. My once-good relationship with my mother, who I never would've considered a "bad person" a few years ago and never displayed any antisocial tendencies to me before, was ruined by her repeated and pervasive deception and theft.

> That is my maybe my main issue with the AI art bots, do the artists who work with them even feel vulnerable about their work?

Sometimes, yes. John Q Nerd playing around with it might not, but the "good AI artists" I follow on Twitter certainly do.

> Can artistic sincerity and enthusiasm exist in AI art?

Unequivocally yes.

> Do AI artists feel the literally metaphysical stakes of what they are doing like Van Gogh did?

Some of them seem to, though I dunno precisely what Van Gogh felt so I guess I'm not sure.

> Is it all just different campaign posters for the cause of legitimizing itself as "real" art?

Not sure where you find this stuff, but it sounds really boring. Certainly not all of it is.

Your unsolicited advice falls flat (to me), sorry. My unsolicited advice to you would be to psychologize less.

Yes, whenever I spend hours refining a prompt in my stable diffusion UI and select one of hundreds or thousands of outputs and spend even more hours refining it with inpainting and other non-AI tools, I've felt nothing whatsoever, and my choices aren't informed in any way by anything interesting going on in my inner life.

You ever feel like this specific propaganda war is actually unwinnable? Many people are extremely motivated to bullshit the public (usually sincerely though I kind of doubt it in this case), and from I've seen, the public are far more willing to believe the 3 extremely online artists who they've heard an opinion on the topic from than the 1 software engineer/data scientist who actually knows half a thing about machine learning they've heard an opinion on the topic from, let alone the growing cornucopia papers and high-production-value websites that seem to say "it's just a plagiarism machine" if you don't know anything about the subject vs the approximately one website I've ever seen that says "no, you are being lied to".

I'd like to believe this isn't one of those things where we can only move on by everyone who believes the various correlated falsities dying, but I don't think I can.

Not to mention that the totality of all other images is in most cases probably more "influential" than the few most visually similar images! Consider the thought experiment:

1. Take the prompt you used, and use it with a model checkpoint that was trained identically to whatever model you're using, except that the top 21 images this website shows you are removed. In most cases, while your outputs won't be identical (I assume), you can probably get something pretty similar.

2. Now, take that same prompt, and use it with a model checkpoint that was only trained on the top 21 images this website shows you. (AFAIK you can't really do this because Stability hasn't released a "completely untrained" version of any of their models... though maybe they have and nobody cares because it's useless for most purposes.) I'm not completely sure what you'd get, but my bet would be that you get either nonsense or a memorized replica of one of the training images, not the same output image you got previously.

Yeah, it's bullshit, but digging into a specific point from their FAQ:

> Usually, the image the model creates doesn’t exist in its training data - it’s new - but because of the training process, the most influential images are the most visually similar ones, especially in the details.

Would be cool if this were true, but I don't think it is, because the prompt you used and the captions on the training images are being completely ignored. If two different words tend to be used in captions for very visually similar images, and you use just one of those words in your inference prompt, I'm pretty sure the images that were captioned with the word you used are much more "influential" on your output than the images that were captioned with the word you didn't use. (Like, "equestrian" vs "mountie" or "cowboy" or something.)

Sorry, but you're reading into noise. Anyone can reproduce an image anyone else made by only knowing the model checkpoint, positive and negative prompt, seed, sampler and sampling steps, &c &c they used. (Well, in principle, and usually in practice too. Interfaces might give different results now compared to a version from a few months ago because implementations of certain things changed, or if you use xformers then all your outputs are slightly non-deterministic, other exceptions that prove the rule like that.)

Some prompts I've come up with generate excellent and definitely novel results (without necessarily much work put into refining the prompt), others are extremely hard to get working well with hours of work even if I know it's something that isn't novel.

Who's "we"? :)

To the extent this perception exists -- and I don't think "came from a computer" falls within the top 5 actually effective methods of laundering bullshit nowadays, though maybe it used to -- you might expect that it gets crushed into dust as the public gets more exposure to high-profile counterexamples.

And, wait, isn't the concern usually that people read AI-generated content and trust it but don't think it came from a computer?

Just imagine the hell we'd be in if people could give wrong answers in an authoritative tone and then insist they absolutely can't be wrong!

At this point, I truly don't know which of your(?) current type of applied arts vs my current type of software development will turn out to be more sensitive to technological unemployment, or on what margins and time scales. So, hopefully this doesn't strike you as callous, since I think it applies to me as much as it does to you or your coworkers:

Roughly everyone who makes money has the same job, which is creating value in expectation for someone else. (Whether that activity is net good for society is a different question, I lean toward usually yes for stuff you can do legally, not always though). If you find yourself suddenly unable to give anyone a competitive deal on whatever expected value you know how to create, because of technological developments or otherwise, well, you'd better figure out what you need to change about what you're offering so you can. I wouldn't call this fair, exactly -- maybe it would be if you or your government had effective "technological unemployment insurance" -- but I struggle to imagine any substantially different state of affairs that's clearly better for the world. (No points for saying "imagine communism then".)