GeneralMayhem

Because Democrats, being (at the moment, and relatively speaking) the party of government actually functioning according to laws, are extremely unlikely to do this in the other direction. The two sides are not the same. It is not possible to be the party aligned with institutions, rule of law, and predictability, while also using the power of government arbitrarily and vindictively. On the other hand, it's entirely possible to use the power of government arbitrarily when you're the party of greed and corruption. So "Republican-aligned companies" don't have to be nearly as worried, and otherwise-neutral companies will become Republican-aligned out of self-preservation, because one side is going to punish them for disobedience (legally or illegally) and the other side isn't.

There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.

The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.

What other solution do you suggest to the coordination problem?

Not OP, but I think they're way more essential with AI doing a lot of the coding. The biggest thing that AI, even the frontier models, is not great at is staying on topic and actually finishing a project with reasonable priorities instead of ratholing on insignificant details or claiming it's "finished" when it's half done.

The most important thing that a good design doc does is specify what's in and out of scope. The second most important thing is to precisely define common vocabulary - what are the important concepts in the problem you're solving, and how should they relate to each other? All of that information serves to ground the day-to-day work in what's important. I find myself starting every Claude session with "read this doc and get familiar with the world, then we'll get to work on a part of it".

(The same is true when working with humans, especially but not limited to junior engineers who aren't used to managing a project longer than a week or two. AI coding agents just never grow out of that phase.)

There is a specific, unilateral action that he is taking described in the article.

And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.

Yes, that is exactly Dario's concern. Either one of the US labs or one of the Chinese ones will eventually release something with insufficient safety controls for its power level because it gives them slightly better user retention (look how much complaining there is about current frontier models, especially Fable, rejecting requests). Regulation or consortium is how you avoid the prisoner's dilemma.

Sure, I acknowledge that it was incredibly rare and lucky for "the thing I enjoy doing" and "the thing that pays" to have been the same for so long. That doesn't mean it's not painful for that to change out from under me - especially when I can't quite convince myself that the new way of doing things is actually better by any metric.

What supply chain controls can you have on biotech? Part of the problem is that you can do a lot of damage with ingredients you can buy over the counter or make at home.

Except that there are finite hours in the day. I enjoy the process and craft very much, but I also enjoy making high tech salaries. Up until a year or so ago, those weren't contradictory, because the craft paid well. Now, continuing to be employed in an AI-dominated industry means that I can no longer practice my craft at work most days. So now I need to find the motivation to do it in my spare time - and this while the work hours are significantly more emotionally and motivationally draining because it's less fulfilling.

If I switched to spending more time on the hobbyist craft, I would pretty quickly have to give up doing well at the job. And the parts of the craft I enjoyed the most are the puzzles around distributed systems and high-scale correctness, which tend not to come up in the hobbyist context anyway.

Cynicism is fine. They don't have to believe in any sort of ideal to do this. They just have to be thinking more than a couple months ahead to realize the math favors quality.

Related in that some of the same organized groups tend to be carrying out every kind of attack at once, but operationally a different thing.

In the ad marketplace there are four participants: the advertiser, the user, the network (Google), and the publisher (also Google for AdWords, other websites for AdSense and so on, and effectively the channel owner for YouTube).

In the click spam or botnet case, the bad actor is the user, who is usually associated with a publisher trying to get extra money (although not always - there's reasons like auction manipulation where some advertisers run click bots too). In the bad-ads case, the bad actor is the advertiser.

At Google, each of those problems has their own well funded team, but they do also share some data to help catch rings of bad guys.

I don't understand what point you're making.

Your "point 1" is literally the argument of the paper. If that scenario arises, the market collapses and no further sales can be made. That's the whole problem. As someone who takes a percentage of every sale, you want to keep the lemon-sellers out even though in the short run they make you extra money.

Your "point 2", if it's meant to be a rebuttal, isn't much of one. Buyers don't need to accurately know exactly what fraction of sellers are fraudulent; if they believe that it's 50-50, the same thing happens, even if the true rate is 80-20 in favor of good sellers. Conversely, if buyers are overly optimistic about quality, the market can persist despite a level of fraud that's higher than should be tolerated. But in any case, things like reviews and external reporting should eventually give them good information.

They come with renewal expectations, assuming the situation hasn't changed. Reserving the right to change the rules unilaterally doesn't make you not an asshole if you actually exercise that right. And government only functions if it's predictable - if it's a random-number generator (outside of processes that are explicitly lotteries), what is anyone supposed to do? Why would anyone engage with it?

In most states, employers have broad rights to fire employees at will, including no reason or "he wore the wrong color shirt". That doesn't mean that if you have a dickhead manager who fires you because he doesn't like the color of your shirt isn't a dickhead, and if you defended that manager I would similarly consider you unfit to participate in civil society.