thoughtpeddler

Amazon did not kill Barnes and Noble. For a moment, it looked like it did. But B&N is experiencing a resurgence the last several years, because humans don't just value getting books delivered the next day through e.g. Prime. They want to read books in a physical space, and will inconvenience themselves for that tactile experience. There is an important lesson in this when thinking about the AI future.

Fascinated by the psychology here. Is it because you view frontier AI labs as beneficial entities that truly care about everyone's well-being, and will do what it takes to ensure, post-replacement, that we are 'taken care of', broadly speaking? Or are you more like "I'm ready to be replaced, and if that results in my death, I was over this reality in the first place, so it just speeds up what I've been hankering for anyway."

How does this follow? Isn't it fair to say that what someone's interacting with at the 'personality surface'-level is a form of emergent complexity / emergent behavior, not readily apparent or discernible at the neuron level (just like how human 'thinking' or 'consciousness' seems to work)? I'm not even arguing these models are conscious, I'm just saying that it doesn't seem like we can take the fact that a single neuron flipping their behavior or triggering 'alter egos' is enough to say they don't "think", etc. We might discover that flipping - if not a single neuron but perhaps really tiny clusters of neurons/synapses - in a human brain, might result in similar effects. We haven't had human connectomics studies to really show this (to my knowledge).

Ambition has to match the problem space. Sometimes I stare at the Bay Bridge here in San Francisco, and I wonder to myself, as an exercise of "how much more ambitious could AI make us?" imagining a 10x version.. would it simply be a bridge 10x bigger? Is that the best I could come up with though? And why is the bridge there at all? ~15,000 years ago you could have walked from SF to Oakland across a dry valley floor (that is now the Bay, true story). Then the glaciers melted, water flooded in, and only then did crossing it become a problem worth solving: first with boats, then with a bridge, and eventually with BART’s Transbay Tube, which isn’t a bigger bridge at all.

Bridges exist to move people and goods across water within a particular envelope of needs and constraints. In this specific case, when the needs grew, the most ambitious answer was not a bigger bridge, but BART's Transbay Tube, which is an entirely different kind of thing.

So when I reflect on Dan's piece, it seems we're in agreement that AI is not provoking us to simply build bigger versions of things we already have.. but what remains unsettled is whether we know what we’re really building for yet.

There's of course the by-now oft-repeated story of electrification of factories, which had a 'first pass' of mostly swapping out the steam engine with one big motor, and otherwise preserving the same layout. Over time people redesigned floor plans around small motors at each machine (with Paul David’s “The Dynamo and the Computer” being the classic telling of this).

A lot of AI-assisted software development right now still feels like the one-big-motor pass at this. I suspect the projects really worth being ambitious about are probably the ones that only make sense once the entire factory floor changes, and I don’t think we're able to see that just yet. What Dan's doing is probably the right answer at the moment, building a lot of odds things and seeing what sticks.

Back to the Bay story, I think this moment is equivalent to our digital glaciers melting, and the water rushing in, than us having to figure out better "cranes", for example. It's changing what the world we're designing for is, what the whole problem space is, instead of whether we build bigger versions of things within the prior paradigm.

We look to our software stack today and we are tempted to imagine 'bigger', 'better', 'faster', 'more efficient', but... why? What are we really building _for_? We shall see!

The US government is not a fan of other sovereign nations (even Western-aligned ones) from gaining "algorithmic sovereignty", as was seen when Australia tried to do so recently and received a stern threat from the US State Department.

I believe that in the fullness of time we will see that by the mid/late-2020s our society was completely information-saturated/overloaded, and hence why turning to these personal AI agents was seen by most as an easy decision (if not downright necessary to gain a feeling of control again over their lives).

Most peers (Millennial knowledge workers) I talk to these days constantly discuss the feeling of "being overwhelmed", and I suspect a lot of it is just the info-sphere.

Neil Postman predicted some of this many decades ago in "Informing Ourselves to Death", where he argued that increasing the speed, volume, and accessibility of information would not necessarily improve society (i.e. information without context/judgment and institutional scaffolding can become socially destructive).

I really feel like we live in that world, and hence the rise of these AI agents as a potential "solution" to that.

Yes, exactly. I remember when the idea of human biological cloning was brought up some decades ago, and society collectively decided "No". Fast-forward to today and digital cloning is suddenly possible and there's absolutely no questioning it - everyone's on board. I'm surprised.

If their security was so poor, why haven't frontier lab weights leaked yet?

Yeah, basically. I suppose as a business owner ideally one tries to walk a tight rope between the 'artistry' and 'fun' of aspects of the business and running it, vs. the cold brutal logic of capitalism, markets, demand, etc. It is interesting that biology is similar: you can as a being of emergent complexity choose to do so many interesting and artistic and fun and social things with your mind, but then you are embodied as trillions of cells, which will eventually succumb to the unfeeling cruelty of natural systems. That's reality it seems!

I'm not saying the 'Claude says this is bad so therefore it's bad' is at the heart of this PHP vuln, but I'm also seeing the same phenomenon in workplace settings with 'meatproxy'-behavior of so much AI use that tends to overfit and hyperfocus on narrow issues, that then punts the work to other people in ways that just waste time (whereby pre-AI, humans would simply not have the capacity to investigate minuscule issues so they'd never be reported as such). But these days, it's just hardcore meatproxy and reverse centaur vibes all over the place (people just acting as the face of AI, and creating so much work that the other people can't really keep up with any of it anymore), and I wouldn't be surprised if that's happening to some extent in infosec more broadly too. (not saying the vulns and exploits aren't real - but there's definitely a long tail of them now where only a few are deserving of a lot of attention, patching, mitigations etc).

I actually rather like 'SOCMED', given that social networks these days can be weaponized in various ways, so it deserves a name akin to 'OPSEC' and other military-like terms

> My north star for Foursquare’s culture (2010s) was always, “We want people to look back and think, ‘That was the best job I ever had because those were the best people I ever worked with’.” And I hear from ex-Foursquare people over and over again that we absolutely nailed this.

Reminds me of what we've been hearing recently from the mathematics community: that if AI completely overtakes their field, it's still about doing math with your fellow humans (just like how people still massively enjoy chess despite AI's dominance over that game).

More and more, technocapital progress reveals the social and communitarian heart of our soul.