hackinthebochs

Progress fetishism will be the end of us all. There is a point where human prosperity is not further served by more "progress". I don't claim to know where that point is. But simply treating progress as some universal good leads to disaster. But it is clear to me that human disempowerment is bad for everyone.

My worry isn't about being made obsolete, it's about human disempowerment. Of course you'll just say AI is a tool and so forth. My point is about where things are going, not where they are currently. AI boosters are rarely willing to extrapolate progress into the future in its defense. It's either willful ignorance or strategic feigned ignorance.

When you need a football field of GPU to host frontier models and advance the state of the art, it's not too late. The supply of GPUs can be monitored and controlled. Access can be limited. International agreements to stop further progress can be hashed out (the China bogeyman is a mirage). We could outlaw the commercialization of more capable models and seize datacenters that aren't in compliance. If the current models were as far as we got, we'd be mostly fine. The only thing stopping us from realizing this future is the inability to coordinate.

There's nothing any of us can do about it individually. But collectively, we still have power. It's a matter of organizing into collective action. This is a political issue just like any other. When the vast majority of people don't want something to happen, we have the power to stop it if we act. I'm not the one to make that happen; there are much better people for that. What I'm saying is how we speak about the issue should be one of agency, not of resignation. Agentic attitudes are contagious, just as defeatist attitudes are. Perhaps collective action on a small scale can spontaneously organize out of it. Or the right people might see enough people poised to act to be the organizing force. None of this happens if we admit defeat before shots are even fired.

It's not too late to do something about it. Acknowledging that we are not powerless to determine what our future looks like is the first step to taking control. We all seem resigned to the impending disempowerment and crisis of meaning for humanity as a whole. But this is just playing into their hands. We should collectively exercise our power of self-determination while we still have it.

It is a very narrow mind that thinks everyone secretly thinks exactly as they do

>and the end user/society at large captures the lions share of the value as real work done.

This is a fallacy where you extrapolate the large change in a single dimension and imagine everything else staying mostly the same, ignoring the disruption of the system from non-linear effects of that single dimension being saturated. When a few companies capture a large proportion of the economic value of the system, the economy simply doesn't exist as it is currently conceived.

The current economy is driven by the need for broad participation to maintain the collective standard of living. We all get to skim a bit of that value for ourselves, and so distribution is inherent to the system. When broad labor participation is no longer needed to create collective wealth, the leverage of the average person to demand a share of the value produced evaporates. The economy in this scenario is something very different. There's no reason to imagine wealth being distributed in an egalitarian way; it certainly won't be automatic.

To understand my point one must separate the metaphysics from the practical effects of the belief system. Christianity represented a step-change towards humanism. This must be considered in the in the era it was founded and compared to the competing ideologies at the time, i.e. paganism and its various death cults. Christianity institutionalized ideals like equality of men, the sanctity of life, charity, self sacrifice, etc. These ideals are the precursors to civilization.

The particular ideology of Christianity was super effective at laying the groundwork for modern civilization. Of course, these ideals have since evolved and have since become some kind of secular ideology. Granted, it was imprecise to attribute a fear of death to Christianity. But the sanctity of life is in practice a close cousin of a fear of death. Modern secularism having no metaphysical alternative has made this fear explicit.

The Christian influence on the world has been undeniably positive. Seeing death and suffering as the enemy and organizing society around minimizing it has been a boon to human progress. But now we come to the dark side of this view. A fear of death so complete that we are willing to sacrifice everything of value to prevent it. Our society has become so divorced from our natural state that death is seen less as an inevitable part of life, and more as a corporeal enemy that must be resisted at all costs. As it turns out, too much of a good thing, whatever it is, leads to absurdities and atrocities.

No, I do not think human autonomy should be sacrificed to defeat death. I personally do not fear death. It's as irrational to fear as it is to fear a deep sleep. I do fear dying painfully. Tragically, we as a society resist alleviating the pain associated with the process of dying (by speeding it up) for the same reasons. If only we could learn to take our principles in moderate doses.

>More importantly, I'm curious how you define an ideal human society. How do you land on what ideal is, or what it would take to get there?

The question is in what context does the human psyche thrive? We find meaning in personal struggle and overcoming obstacles. We're happiest when we're striving for something out of reach but seemingly attainable. We're much less happy when we have all of our needs met but have no source of meaning or fulfilling struggle. Our evolutionary history is one of struggling against hardship and finding personal meaning in the process. This is a core feature of our psyche and it is to our detriment to eliminate it.

This isn't to say we should all be struggling for our next meal. Much like cats, our core drives can be exercised and satiated through a simulacrum. There are many ways it can be satisfied, but strive we must, against something with inherent meaning to us. The problem with AI fully realized is that it takes away most avenues for human achievement, thus the struggle loses all meaning.

The ideal human society is not one where people are narrowly focused on creating widgets or increasing efficiency, narrowly construed. An ideal human society includes intellectual pursuits. Some of these pursuits do not suit dabblers, they take deep expertise that can only happen from a lifetime of directed effort. We are collectively better off when some value is spent in ways that isn't aimed at maximum efficient allocation.

It's funny, the justification for accelerationism is that people can be paid (UBI) to pursue their passions. On the other hand, we have those that are asking why anyone should be paid for work that isn't economically viable. I suspect we're being sold a load of bullshit.

>However, I do believe that when a couple problems get solved that massively benefit all of humanity — e.g. a massive cancer breakthrough, etc — a tipping point will be reached and public opinion will reverse and enthusiastically embrace AI, and will accept not only diminished personal meaning but outright technocratic rule.

Is solving cancer worth ceding human meaning and human autonomy to the machines? I haven't seen anyone ask this question but it seems the central issue of this revolution. Personally, I don't think it is.

Perhaps machines will lead to a perfectly homogenous society completely absent of negative valence whatsoever. But if every experience is also devoid of any personal meaning, what's the point? The optimal human state is not one of peak hedonism, or wholly absent of suffering.

Only those that aren't in a field and see its existence justified for purely instrumental reasons can think its value is merely in its practical output. Personal meaning and fulfillment come in many forms. For mathematicians, part of that value is through solving problems, but it's also partly about the struggle itself, gaining understanding, expanding the frontier of human achievement, being a part of a community that values these things, playing a valuable role in the progress of humankind and making a career out of it. Having all that swept away overnight will lead to an existential crisis for anyone who is on the wrong end of this process.

It's quite bizarre how many people feel so personally affronted at the existence of a rather exclusive intellectual community that they take glee in seeing it decimated overnight. But make no mistake, AI is coming for the things you value as well.

>IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.

Whether some trait is emergent shouldn't depend on the state of our knowledge. That no one predicted how far LLMs would go is a demerit for our model of them and doesn't imply they have extremely complex "emergent" interactions. My conception of emergence is asking whether something can be "easily" predicted from an ideal understanding of the component parts. Simple/linear aggregations of the correct semantics of precursor neurons is what is at stake here. In other words, is understanding the semantics of the precursor neurons the vast majority of the hard part in understanding what comes next? Past failures of prediction do not bear on this question much at all.

An example from the article that is informative to my view is the association the network made to the base64 encoding of plain text substrings. If you had asked me, I would have assumed the network decodes base64 live as it traverses the layers. Instead (at least some of) these associations are precomputed during pretraining and recorded in the form of shared affinity in neurons. Active computation and precomputed associations are on two ends of the intelligence spectrum. The human brain is strongly biased towards precomputed associations. It seems that LLMs are as well. The more I see about their inner workings, the more I suspect they operate mostly in the form of these explicit associations. The computations they actively manifest across the network can't be that complicated due to depth constraints. There just aren't enough compute steps[1] for the kinds of complex, non-linear, unpredictable dynamics you get from compute heavy regimes.

>Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc.

When I talk about explaining some phenomena, I mean it in a robust sense that e.g. gives you the capacity to predict downstream behavior with high accuracy.

[1] These discussions always seem to have difficulty distinguish the computation that realizes the system from the computation the system as such engages in. There is a huge amount of compute that realizes the feed forward pass of an LLM. But the computational steps directed by the LLM to resolve its output from the prompt is severely restricted by the layers in the forward pass of the network (or however many virtual layers from looping).

>you must be using some completely idiosyncratic definition of "emergent" here.

I doubt my usage is idiosyncratic. Rather, the term is so ill defined that people often use it inappropriately. Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts. My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons. This isn't something I can prove, but it certainly isn't obviously false from just considering recent mathematical results. Higher order/non-linear interactions are characteristic of this kind of inexplicable behavior from complex interactions we characterize as emergent. But there are good reasons to think this kind of non-linearity in any semantic regime is next to impossible in principle.

Semantic geometric representations are characterized by linearity in response dynamics such that a linear change in one variable results in a linear semantic change in related variables. That's how vector embeddings capture semantic relationships. Call it the linearity hypothesis. Non-linear dynamics in the embedding space just destroys semantic relationships in most cases. There is no way to keep an entire network of relationships sufficiently correlated if non-linear response dynamics is a part of the representational toolbox. Non-linear dynamics in general kills correlations.

Testing this against recent LLM proofs, what would need to be shown is that the relevant neural activations do not decompose the semantic content of the proofs (or the reasoning trace) in a straightforward manner, but rather the reasoning trace is unpredictable from the semantic content of the activated neurons that precede the tokens of the reasoning trace. But this idea strains credulity; it is almost inconceivable that this would be the case. We should be highly incredulous of the idea that complex non-linear interactions can be wholly responsible for the long-running coherent output in LLMs. The default hypothesis should be that there is no emergent behavior (as I defined it) in neural networks.

>Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.

Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.

>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).

I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.