sweezyjeezy

My question is not how do you defend against it, my question is whether ablitteration makes the model completely oblivious that it shouldn't be doing 'the bad thing'. Like if you manually prefilled a normal model's output with that action, it would likely go "oh wow I can't believe I just ran that, I must warn the user of this now..."

Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?

That's not what you said though - you said it was a bad career choice. I'm saying it was a good one, even if you didn't end up directly doing a lot of high-level math.

If your job was previously mainly following instructions from SO, and this has now been replaced by Claude - your employment prospects are not looking good right now. You're just a middle-man between your PM and the codebase that possibly doesn't need to exist for long.

I think you are mistaking 'happiness' with 'employment' here. The leather worker may adapt to the new environment to become a factory site manager. But there's no guarantee that this leads to happiness.

Also you should bear in mind, adaptation is a young man's game. Retraining is possible, but companies typically don't hire trainees in their 40s/50s, so your willingness may not be the deciding factor. My dad ended up as a postal worker when he left his career as a humanities academic at 55, no highly skilled jobs would take him (he did prefer it though haha).

I am an ex-mathematician who went into industry, I would say apart from CompSci/ML and physics, it was one of the most sought-after PhDs you could have (across tech and finance) - when I was entering the job market 10 years ago at least. No regrets.

It feels like it's meant as career advice though. I have no idea what advice I would give to a teenager right now, but as a math PhD - math would not be on the shortlist, sadly.

I actually have a rather dim view of the "writing code was never the point" line. Not because it's objectively wrong, but because I see it as something we're mostly telling ourselves to feel better about the status quo. Ability to write good code has been highly celebrated (and remunerated) for decades. As it is becoming less relevant, we immediately backtrack and start lionizing the parts where we can still be useful instead. Consider the counterfactual - AI continued to be terrible at writing code, but weirdly better at humans at product decisions, architecture etc. In this universe, saying "coding was never the point" would not be popular.

It also find little solace in it aside from 'well this version of GPT isn't taking your job'. AI labs certainly have no intention for the higher level skills to stay in the human-only domain. The veteran developer with the coherent theory of a large stack is immensely valuable today. But they also don't survive if a company can drop a few coders' salaries on rewriting that stack from scratch - faster, fewer bugs, more coherent, able to react to changing business requirements with more agility etc. I am not saying this is where we are, but I think there is a reasonably good chance this is where our road is leading us.

Broadly I agree with this, but I was refuting the statement "there has never been a better time to be a mathematician". You are changing the question to something different here, and using it to say my argument is bad.

I think math is a microcosm for "thought-work" in general. We have been in a symbiotic relationship with capitalism for decades now, where the hope of a well-paid white collar career encourages people to spend time and money to enrich themselves through education. The proliferation of AI cheating at college already signals that employment is the primary goal over intellectual growth, so one imagines this governs what happens next if labor demand disappears.

It's hard to say where this all leads, but I have very low optimism for higher-level math understanding being something that humans value in the same way in the coming decades.

I'd even say Scholze is not a great example here. Most of the work he's known for is progression towards the Langlands program - which is very much a problem to be solved, and one I would imagine he'd not be thrilled for an AI to one-shot. I agree that it is somewhat 'up the chain', in the same way that software engineering has not immediately disappeared now that performing coding tasks is largely automatable.

But I also take issue with 'never been a better time' - e.g. is this really the greatest time to be a software engineer? Everyone has AI psychosis and feels like they're a couple of breakthroughs away from being unemployable. The same is even more true in math - we've gone from failing IMO problem 6 last year, to solving NS. The rate of change is formidable, it feels like there may not be many places to hide in a few years.

> In my opinion there has never been a better time to be a mathemetician...

As an ex-mathematician I assure you this is very wrong, and every working mathematician I know right now is completely miserable, and/or trying to flee the field as fast as possible. It's like telling a chair-maker during the industrial revolution that there had never been a better time for them, since now they could operate chair-making machines instead of toiling away at the wood themselves. It assumes that they were purely in it for their passion for mass-producing chairs. The majority of mathematicians get into the field because they love problem solving, and the gauntlet thrown down by challenging math tasks.

Many parts of this will never be useful for society on a grander scale - but this is reflected in the finances - pure math is closer in funding-terms to a humanity than to hard science. Now even this is _massively_ under threat, and Tao and co need to pivot quickly to stop this from becoming a bloodbath.

The math field is confronting something that coders have been dealing with for a few years now, only far more violently. Today's moat for software seems to be that AI can automate tasks but not a full job (yet). But for a large proportion of mathematicians, doing these tasks really was _the_ job. It's the bit they wanted to do, and if they completed a sufficiently difficult set of tasks, they got tenure. Now this model is failing, they frantically need to pivot the role of humans to save their profession from funding cuts.

I remember when "writing code was never the point" became a mantra here. There was truth in it, but removing the coding has certainly taken away a lot of the texture of the work and enjoyment of the craft. Many of us feel this loss as we tech-lead teams of agents as our source of income. I am not optimistic the mathematics pivot is going to work, but I'm certain that most will be depressed with the outcome even if they succeed.

We are all staring at the same existential dread, just seeing it unfold slower. We're being told that utopia is to be obsolete, and that is a jarring idea to contend with.

As a chess fan, 100% this. We have known for the last ~15 years who the best human chess player is, and that he will lose against stockfish on his phone. But chess survives because of the human characters involved, the rivalries and dramas, watching two people trying to overcome each other under insane pressure, and sometimes coming up with something astonishing. In short - it's a sport.

There is no equivalent in math.