origin_path

Which had very little impact on humanity because it exploded in the middle of the tundra.

"These are predicted to happen once every hundred years or so"

What is predicted exactly, by whom and how were these predictions validated against testable reality given the postulated rareness? If they're so common then why is it so hard to name the last 10? I think in reality these events are very rare and will almost always happen over the oceans, deserts, poles etc where not many people live.

"Long-tail risks exist, and burying your head in the sand and pretending they don't"

They exist and I am not pretending they don't. I am saying that this style of reasoning in which an extremely unlikely event is unfalsifiably and arbitrarily assigned near infinite downsides in order to justify spending time and resources on it, is problematic and as a society we are far too generous towards people who do this.

There are no actual safety issues with LLMs, nor will there be any in the foreseeable future because nobody is using them in any context where such issues may arise. Hence why you're forced to rely on absurd hypotheticals like doctors blindly relying on LLMs for diagnostics without checking anything or thinking about the outputs.

There are honesty/accuracy issues. There are not safety issues. The conflation of "safety" with other unrelated language topics like whether people feel offended, whether something is misinformation or not is a very specific quirk of a very specific subculture in the USA, it's not a widely recognized or accepted redefinition.

This kind of elasticity in language use is the sort of thing that gives AI safety a bad name. You can't take AI research at face value if it's using strange re-definitions of common words.

These two things are in conflict. We could ignore both asteroids and climate change and according to the best known science there'd be very little impact for vast timespans and possibly no impact ever (before humanity is ended by something else like war).

Yes, also for the climate. Look at the actual predictions and it's like a small reduction in GDP growth spread over a very long period of time, and that's assuming the predictions are actually correct when they have a long track record of being not so.

Really stuff like asteroids and climate is a good counter-argument to caring about AI risk. Intellectuals like to hypothesize world-ending cataclysms that only their far sighted expertise can prevent, but whenever these people's predictions get tested against something concrete they seem to invariably end up being wrong. Our society rewards catastrophising far too generously and penalizes being wrong far too little, especially for academics, NGOs etc. It makes people feel or seem smart in the moment, and they can punt the reputational damage from being wrong far into the future (and then pretend they never made those predictions at all or there were mitigating factors).

That's the average wiki. It's a commons and a tragic one. To make docs work you have to treat it more like a codebase: clear ownership, standards, review processes, approvals, up front design, refactoring efforts etc.

Now you're right it's obviously not that black and white, I'm generalizing. But I think devs often under-estimate how many people in a typical company perceive meeting other internal employees as amongst their primary outputs, as an end in and of itself, not just a means.

A good way to observe this in action is to try and enforce a rule that meetings must have pre-published agendas. Good luck with that! People will just work around it or write useless non-agendas because often a meeting is not to get something specific done, but is used more like a sort of coffee break to split up the day and give people something to look forward to between desk time.

Something else worth remarking on - a lot of people in sales or marketing roles never seem to use word processors. They communicate ideas by sending PowerPoint decks around, often with a density of words in the slides too high to actually project (only readable on hi-dpi screens). Where I last worked there were people whose working hours boiled down to meetings and PowerPoints. They could spend a whole week making a deck, which would only be seen by their colleagues in a meeting. I found it odd but maybe the slide templates help them structure their thoughts.

1. Standard interviews don't assess reading/typing speeds. If you want a high documentation culture this is critical. It took way too long for us to figure this out but many people in the company were significantly slower at reading/typing than us; they found long documents overwhelming and would find excuses to not read them. Slack conversations became a massive sore spot because unknown to us some people felt like they couldn't keep up. They'd try to type a question or response and we'd already posted another two paragraphs before they got a chance to finish their thought. They'd complain to each other that if they asked us a question they got back an essay in response, etc.

2. Documentation requires ownership, otherwise it rapidly becomes useless. Standard corp tooling like wikis doesn't make such workflows easy. They are however optimal from a corp politics perspective (dispersal of responsibility). Maintaining markdown based websites works well as long as you have empowered maintainers who view document quality as a core job function, but you have to force people to submit changes by e.g. rejecting at code review time changes that don't update the docs. People will moan. They will ask you to do it for them. They will submit absolutely min-viable docs changes, they will demand you hire technical writers even if they're easily capable of doing it themselves. And of course the moment you're not using a git-style workflow, just forget it, you have no chance of preserving coherency in any sort of knowledge base.

3. Lots of people aren't just slow but actively HATE reading and writing. They will make things up on the spot, or lie, or just flat out refuse to do the work rather than sit down and read a long document. Jeff Bezos has said about why Amazon uses meeting time to force people to read the memo:

"If we don’t, the executives, like high school kids, will try to bluff their way through a meeting"

You will have to fire people for refusing to read things if you're serious about creating and maintaining such a docs-oriented culture, which in practice is so unpleasant nobody ever does it and so maintaining such a culture is nearly impossible. You will also have to flat-out refuse to meet people in order to force them to read, because otherwise they'll receive a document and just ignore it. I had several cases where one of my most senior engineers would assert that a product we used didn't have feature X, and I had to correct him by pointing out that the user manual discussed feature X in detail. I knew this because I'd actually read the user manual cover to cover. Basically nobody does this and guess what, if you're the one person on the team who reads stuff then you're going to come across as the awkward smart alec who makes people look stupid. Sometimes, ignorance is bliss.