deanCommie

Who are the Big 4 AI Labs?

To me there are the clear big 2 that actually seem to be innovating and pushing things forward. Meta almost earned a seat at the table with the open strategy then stagnated. Google obviously is the OG but at this point they seem in the same bucket as Amazon and Microsoft as "Big Enough To Matter and Influence Politics and Spend Billions To Hang Around". But they're nowhere near the same tier as OpenAI and Anthropic.

Oh and of course there's SpaceXAI which is Elon's personal money and reputation laundering corporate entity. Elon took over the "king of the reality distortion field" from Jobs as far as the rest of the industry is concerned, so he'll stay first in line for Jensen's chips and investor dollars until he's dead. But as a company what have they actually accomplished besides making the models with the sense of humour and judgement of 14 year old boys?

Even before AI we used to say if you write code that you only barely understand, then it will be to complicated to debug. (and/or maintain)

Mochizuki was still one human and it required legions of other humans to unpack and untangle to confirm that it didn't lead to anywhere in particular.

AI is now capable of constructions so complex that no human or human team can unpack. And its ability to increase that complexity is growing while our human ability is stagnant.

meta-AI analysis cannot help. We (software professionals who use AI regularly) already know that if you run into a situation where a Fable/Astra-generated analysis reaches the limits of our comprehension/complexity due to their subjectivity, throwing more AI at the problem doesn't always converge.

There are many reasons to feel optimistic about AI, and ultimately its general ability to help science and mathematics.

I see no reason to feel optimistic about the future of mathematics and AI based on the current path of frontier labs, unless the misalignment Tao is writing about can be reconciled.

Look, I get where you're coming from, I do.

In the last decade, we've seen rampant abuses and exploitation of open source projects and of maintainers. So much individual good will and generous efforts wasted because of corporate malfeasanse.

AND YET. If you "open source" a project, if you actually invest in BUILDING a community, you then become a PART Of that community - with the benefits and obligations that come with it.

You don't want the obligations? Fine. But then you don't really get to keep participating in the community. And if you act like "This is MY community, I get to do whatever I want", people are free to shrug, and say "maybe i'll go find a different community".

Which is exactly how I see people reacting to Zig right now. They're seeing that an active member of this community (which, as you said, keeps insisting "he owes them nothing because the community wouldn't exist without them") treats people he doesn't like, and they think "Oof i don't want that to one day be me."

Putting aside that I don't agree with Bram (I've been using all the Claude versions he refers to and haven't experienced this), I do think it's interesting that there is no universally perceived golden sweet spot between "sycophantic" and "rude".

Many neurotypical people call neurodiverse people (software engineers) rude, while they think they're just being direct.

Many neurodiverse people call neurotypical people sycophantic, while they think they're just being polite and friendly.

It also happens across cultures (Eastern European vs. Western European; European vs. North American).

So I can easily imagine that when you have a software tool whose interface is language, but its user base is extremely wide across both cultural lines and neurodiversity spectrum, it's going to be basically impossible to nail a sweet spot.

You make it too friendly, and the nerds get mad. You make it too adverserial, and the normies call it rude.

I wonder what kind of communicator Bram Cohen is. Is he succeptible to this? From what I heard about his career, he's always been more of a solo programmer. Has he had to interact with other humans much giving feedback? Could it be that he asked the model/tweaked his prompts to ensure directness, and now he's interpreting that directness as rudeness?

like with most things, these things are both overrated and underrated.

are there performative jobs|tasks|employees|cultures? yes.

are most of the things that engineers think are performative and useless actually so? nope.

some examples:

* managers managing upward - feels useless - is actually the most impactful bang-for-buck for managers to give their teams space to operate without micromanaging

* sales and marketing. The best software in the world won't get known, bought, or used, without good sales/marketing. There is no meritocracy on quality. Almost no business succeeds through technical credibility alone.

* 1on1s. They may not add any value to you, but 1) you'd miss them when they're gone, 2) i don't know how else you expect managers to stay on top of employee concerns - just know "inately"? 3) they may matter A LOT for your teammates, and them being happy means your team will be happy

There are other things like that.

They're not vulnerable to this.

But first, in the phrase that you quoted, you do understand that in human society "trying to harm" someone is still a malicious act.

If I push a coworker at an office window, and it shatters and they fall to their death, there will certainly be some culpability to the building since the window "Should" have been able to hold the pressure.

But I will still be culpable to this.

Second, the threat of more harm is looming. Does the author know that this kind of prompt injection doesn't work anymore?

Either:

A) If they know, and like the principle of it, then every thread here debating their moral virtue is irrelevant. This is an empty protest that will be ignored by the AI model harnesses, and I simply don't see the value in it.

B) If they don't, because they're so hellbent on fighting AI that they're out of touch with the real capabilities of modern tools, then:

i) They are anti-intellectual and incurious and forming their beliefs on incomplete data, and therefore not credible. But more significantly:

ii) *They are a risk for escalation*. If the author realizes this kind of prompt injection can't work, who's to say they won't try to develop and inject more sophisticated attacks?

You're not wrong, but it's also so sad.

FAANG used to be the _dream_. Change the world. Work on groundbreaking tech. Solve harder problems than anywhere else. Get Paid incredibly well.

Then I guess a generation focused exclusively on that last part flooded the zone. I still believe that The Great Resignation of 2021 did more harm to our industry than COVID or any interest rate or VC changes.

It polluted the brains of most of the people in our industry from a missionary mindset to a mercenary, and it decimated big company's established cultural memory all to prop up a bunch of unicorns who will probably all slowly die over the next decade.

So now half of the people can't get a job, and the half that can are miserable. This was true BEFORE AI.

When I was making 200k, 120k of that was cash.

When I was making 800k, 200k of that was cash.

Like I didn't discount that it gave me the ability to buy a house.

But day to day, week to week, it's the same doordash and the same restaurants, and the same vacations.

It's not like I started getting invited to St. Barths.

My sentence:

> other than the ability to buy a house or something, and feel safe with a retirement nest egg

Your sentence:

> 800k is the ability to buy a house and support a family on a single income

So we agree.

The thing you missed in my post:

> day to day

Day to day it doesnt feel different when you own a house vs rent one.

Day to day you're eating the same food, entertaining the same ways. Doing the same thing with your friends, family, and loved ones.

The 50% of your waking life that's not your job is not meaningfully different.

So when the other 50% starts to feel dramatically different, you notice.

Every big tech company's embrace of AI is making all of their employees miserable.

Whereas if you're half-competent and at a startup, the AI is an incredible opportunity to try to leap ahead while the prices are subsidized (by the big tech behemoths fighting wth each other)

The reason is a complete inversion of Ownership and Agency.

For a decade of ZIRP, big tech convinced its employees that they're "changing the world", and what we did mattered. Sure the exhorbitant salaries and constantly rising stock value didn't hurt, but honestly other than the FIRE cultists, for most of us the difference between 200k/year and 800k/year didn't feel much day to day (other than the ability to buy a house or something, and feel safe with a retirement nest egg). No, most people were missionaries not mercanaries.

2021 was the first crack. The comps went crazy, half the industry turned over, and the ones who didn't felt a bitter sting where it became blatantly clear that all the new arrivals were just in it for the $$$, and the companies were willing to pay for the backfills but not to reward the loyalty of the missionaries.

Then came the yearly layoffs, chipping away further, and reminding every employee that they're at the mercy of a spreadsheet and the whims of people 3 levels above them in the org chart, in spite of the economic reality of their product, or their personal productivity.

And now we're here, and it's clear that all of the above is still relevant. The old-timers that hung around see that their personal output doesn't matter, their product's PnL doesn't matter. All that matters is 1) the company's AI strategy (and if they're not part of it, they're secondary), and 2) tokenmaxing.

How can anyone find joy in this environment unless they're purely in it for the comp?

I couldn't. I left my big tech job in December after 15 years, and have not been this happy at work since pre-COVID.

The Three Mile Island accident is also dramatically exaggerated in the public conscience with its severity and risk factor - solely because of the default fear of Nuclear.

Oil, Gas, Coal, and random chemical plants have had much more significant accidents even in the US, but never made a blip in the public's minds.

Aren't France and Canada the ones to learn from at this point with regards to safe nuclear operation?

This is the first fundamental flaw of the article

> Bob's weekly updates to his supervisor were indistinguishable from Alice's. The questions were similar. The progress was similar. The trajectory, from the outside, was identical.

No they won't be. They might be worse. They might be better. But they'll be very different.

And, like you said...

> Alice and Bob had the same year. One paper each.

No they won't. Alice would've taken a year. Bob would've taken a few days.

You've already covered why that might actually be OK, so I'll talk about the author's other error:

> This sounds idealistic until you think about what astrophysics actually is. Nobody's life depends on the precise value of the Hubble constant. No policy changes if the age of the Universe turns out to be 13.77 billion years instead of 13.79. Unlike medicine, where a cure for Alzheimer's would be invaluable regardless of whether a human or an AI discovered it, astrophysics has no clinical output. The results, in a strict practical sense, don't matter. What matters is the process of getting them: the development and application of methods, the training of minds, the creation of people who know how to think about hard problems. If you hand that process to a machine, you haven't accelerated science. You've removed the only part of it that anyone actually needed.

Keep asking why. why does the development of the application and methods, the training of minds matter?

the goal isn't abstract. The goal is still ultimately for the benefit of humanity, just like the cure for Alzheimer's.

Humanity learned physics, so we made rockets and now we have satellites, and the entire planet is connected with communication and information.

Humanity must continue to invest in astrophysics so that we do not get wiped out by a single rogue asteroid barreling through the cosmos, like the dinosaurs did.

Now i'm not saying that there isn't other benefit to making generically intelligent humans that know how to think. But at the end of the day, the purpose of astrophysics is no less existential than the purpose for developing medicine.

I want to know the age of the universe so that we can understand what created it, and if we can reverse entropy, and if there is anything beyond the universe. That is a quest for humanity that will take hundreds if not millions of years.