topicalSoup

If I might offer some criticism, allowing an LLM to write all of your site copy, especially documentation, is a serious hindrance to comprehension. LLMs are getting better at a lot of things, but as of today they are still terrible at writing concise and effective documentation. If you really care about this project, you should scrap all of the AI-generated text and write how it works in your own words.

I don’t understand why this idea that it’s all just marketing is so prevalent on HN. It doesn’t make any sense. Why would a private company attempt to convince everyone that what it’s building is dangerous to its own customers?Can you think of a single other time in American history where an entire industry was calling to be regulated? I don’t seem to recall Facebook asking for tighter restrictions on social media. I don’t remember Shell telling the federal government that they need to act fast on climate change. Companies do not market themselves by convincing the public that their products might literally kill them all.

The only way this behavior makes sense is if the AI labs believe two things: one, that what they and the other labs are building is legitimately dangerous, and, two, that if they personally stop building then another lab will simply continue (and that lab will be, simply by virtue of not stopping, less safety-concerned than them). This is why they are calling for regulation. Competitive pressure and investor obligations prevents them from unilaterally stopping development. Government regulation is the only (and the most the appropriate) avenue for slowing down AI acceleration.

I see this kind of opinion being expressed on HN quite a lot, and while I understand why people might cynically reach that conclusion, I actually find the simpler answer here more persuasive: the CEOs are being genuinely cautious.

If you look at the AI space, most of the big players (as in the individual researchers and engineers) are getting worried. The ideas that circulate on LessWrong have become more and more influential and in vogue in Silicon Valley, and people are scared. You should listen to Jacob Coxon’s interview with the New York Times. He describes how working on this kind of AI research had induced a sort of insanity in a lot of his past friends and coworkers as they begin to grapple with the implications of what they’re creating.

Sometimes, HN can be a little too skeptical and anti-corporate for its own good. I personally believe that concerns around AI risks are well-founded, anywhere on the spectrum from mass labor displacement to actual harm to humanity. I rarely see people here actually wrestle with those ideas, instead defaulting to dismissing LLMs as too stupid because Claude wasn’t able to write as elegant of a compiler as them. AI has clearly improved radically in the past few years, and it’s worth extrapolating that into the future and taking a good hard look at what that means for humanity.

Wouldn’t Occam’s Razor clearly point towards the most likely answer being that the sandboxes were simply accidentally misconfigured and models that are very good at goal-seeking did what they were designed to do?

The alternative explanation requires a conspiracy and an enormous amount of lying to the public. Is it possible? Certainly. But Occam’s Razor would not point us in this direction because conspiracy is a more complicated explanation, not a less complicated one.