Joined 5 years ago
Stats like this get thrown around a lot but it's worth considering that the ABSOLUTE numbers might not actually change that much over time.
With agriculture specifically, we can make some rough estimates that, before mechanization (say, 1760-1800 period), there was maybe 10^9 people on Earth and they were all employed in agriculture. So ~10^9 farm-workers.
As of 2024, FAO estimates of global agricultural workforce put the total at about 892 million; call it ~9x10^8. [1]
So, more precise (but perhaps not much more accurate) to say that mechanization put 10% of farm workers out of a job, not 99%.
Now, it certainly IS true that mechanization allows (or forces, depending on your perspective) 90% of the population of Earth to do something other than farming, which is the real point you were making - Technology DOES have a way of disrupting what we've been doing for generations.
[1] https://www.fao.org/statistics/highlights-archive/highlights...
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
They might be! Here's one extraordinarily simplistic argument for that case:
1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies.
2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved.
3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI."
4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either.
And so a dilemma:
- If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race.
- If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either.
I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests.
But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing.
In other words... "We must pursue advancements in AI to protect us against advancements in AI?"
edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me:
1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before?
2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"
"Being outmatched" is not a novel situation for humans either individually or collectively and there are a hell of a lot of ways we can approach that situation productively. OpenAI doesn't appear to have bothered.
Here's a freebie: if you're building something that might turn out to be Skynet and you don't know what it's capable of, your testing regime should assume it is capable of doing bad and unexpected things and account for that possibility: airgap if you can, monitor all network traffic, monitor all hardware usage statistics, log everything, constantly analyze logs, collect baselines and snapshots, also don't trust anything from a device that a model is running on without cross-correlating with other information as much as possible (does your AI inference server claim low utilization? put a temperature probe on it and see if it's staying cool or getting hot, maybe Skynet-Alpha is overwriting /proc to mislead you for reasons you don't yet understand!)
In other words, if you WANT to be able to nip things in the bud - buy some nippers and watch for buds. Whatever else this situation is, or may turn out to be, it is not a situation where OpenAI was on their guard and still got surprised.
On the other hand, I don't want to be charitable. OpenAI very nearly couldn't have done this "research" worse if they tried - the list in the linked article starting with "While we are here, it’s worth listing the other top holy shit moments" is genuinely jawdropping. What were the humans doing in all this? Nothing, or worse than nothing eg. point 1 where they saw the message board and didn't consider it something to escalate internally.
If you take this information at face value, it's as though OpenAI did not take seriously the possibility that something like this could happen, since they took absolutely no steps to prevent it.
Or perhaps this is "normalization of deviance" that's leaked out into the public sphere i.e. they have research teams seeing this kind of behavior all the time internally and they've gotten used to it, "of course agents come up with a collaboration mechanism when given the chance, what else is new?"
To your point:
> Isn't this a testable hypothesis? If consciousness spread virally and isn't a natural outgrowth of the physical structure of the brain, then presumably isolated indigenous tribes would still have pre-consciousness mental processes, and that should be detectable somehow.
SSC's article points to a Reddit thread that suggests even folks in the mainstream of culture may undergo a "consciousness transition" as adults: https://www.reddit.com/r/self/comments/3yrw2i/i_never_though...
Not just five times the peak demand, but nearly 3x the total capacity in Texas and one-third the total US capacity in 2025 [1]. Compare that to the 11 GW of capacity "under construction" for Texas in 2025; will look for 2026 figures if they can be found, but I have a strong suspicion that there will still be a mismatch between demand and supply even now.
[1] https://www.publicpower.org/system/files/documents/Americas-...
Seriously though, this is a clever idea and I'm interested to see if I can consistently beat the monkeys. Time will tell! Thank you for sharing!
> ... if he sells a new oven to the country’s pizza makers, pastry chefs, and bakers, he only needs to capture 10% of the market to become a billionaire.
i.e. the founder assumes that "ovens" is a single $10b-sized market, and it isn't, because reality has a surprising amount of detail (as sibling comment also notes).
You could interpret this whole story as a saga of very inefficient oven market(s) research, the end result of which is the forum post at the end:
> On the forum an old user warns “Make sure that you support rotating bases day 1”.
"Technical" and "cognitive" debt aren't really distinct phenomena; the spirit of the original definition of "technical debt" was that it WAS the delta between the system-as-it-is, and the human understanding of how best to solve whatever problem the system was intended to solve [1].
If we accept collapsing them back down to one term, then "managing cognitive debt" is the same thing as "managing technical debt": work to match the system to the human understanding of the problem the system is meant to address. The article calls out "emerging" techniques to do just this:
- More rigorous review practices
- Writing tests that capture intent
- Updating design documents continuously
- Treating prototypes as disposable
To me these are not "emerging," but rather "well-known industry best practices." Though maybe they're not that well known in fact? [EDIT TO ADD] On the other hand, it would make sense that they ARE well known, and that teams therefore reach for these familiar techniques to try and solve this "new" problem.
Putting in my 2c for the closing questions/thoughts in the article:
> How will they shape socio-technical practices and tools to externalize intent and sustain shared understanding?
Honestly? We'll probably end up doing these things more or less the same ways we always have. AI has not actually changed anything fundamental about how an individual encounters the world; there always was, and always will be, and always will have been, WAY more going on that we can fully get our heads around, but it's also always been the case that we can partially get our heads around most any problem space
> How will they use Generative and Agentic AI not only to accelerate code production, but to maintain their collective theory?
I suspect the answer to this one might well be that high-performing teams will have to scrupulously AVOID "accelerating code production" using AI in order to make sure what they are creating actually composes into the system they think they're building. If human understanding is the bottleneck, then the humans will have to produce less crap they need to understand!
[1]: https://wiki.c2.com/?WardExplainsDebtMetaphor, particularly the "Burden" and "Agility" sections.
Or you sold something to a non-American entity in a dollar-based market, eg. oil. The dollars do come from America to begin with, but once they get "out there" they work as a medium of exchange for whoever wants to use them for that purpose.
In another sense, you can't *avoid* that world; the world they lived in was one they *created*, physically, and much of it is still here with us, shaping us as they shaped it.
And remember, none of the people who came before us ever experienced anything but pieces of their world, just like we only ever experience pieces of our own. But you can at least try to show your kids as many of those pieces as possible.
