rakejake

This precisely - AI takes away your leverage as a software engineer. AI models are very well trained at coding today so I'm not sure what further improvement really looks like. But even if humans stick around to guide the AI, I expect the job itself will get more unpleasant because capital/management have no reason to treat this labor class well anymore.

+1. It is my favorite Radiohead album and the one that I revisit the most.

Where I work, PR counts are a major metric come perf reviews. Surprise, surprise every team has a pointless migration going on. The number of promotions though...

This is a great question. I am not a mathematician but I've had similar questions about my own field (CS/Elec) as well.

I have always felt that a huge amount of brain power has to be expended in understanding the jargon and getting used to the machinery. Once we get past that, what people would think are highly challenging subjects become rather simple and logical. An analogy I like to use is old people struggling with simple tech tasks. I remember an uncle of mine who approached me to fix their laptop which wasn't connecting to the internet. Turns out the laptop had not been booted in a while and the public certs shipped with the machine had expired. Said person is a highly accomplished person in their field but was helpless when it came to this task. After I solved the issue, the uncle was eager to know what the problem was and I tried to ELI5 it but to no avail. It is possible I did a really bad job of explaining it but there was too much basic machinery to cover. Why is there a public and private key, what is symmetric/asymmetric, why do certs even expire etc. I couldn't even touch on the foundational math underpinning all of it so I left him in a confused daze.

To any SWE, all this is table stakes but for non-tech people it is way too much jargon. Today, I'm confident my uncle could prompt his way to a working understanding of PKI - heck maybe even some aspects of cryptography and prime number math that makes it work. This is where I see LLMs making a massive difference - they can crunch a massive amount of jargon and leave only the actual trick/idea for you to understand.

This is also what people like Szegedy and Tao have also said - LLMs will make math closer to what SWE is - a lot more people doing it in workmanlike fashion rather than a small community writing papers. At one point, dynamic programming was some research topic that Bellman worked on and now every 20 year old works on tens to hundreds of these problems in the hopes of cracking a job. But I'm sure the Math field will go through some turbulence and I wish them the best. There will always be a need for SMEs who deeply understand a subject and can come up with interesting problems.

Tbf OP said they have the tools to understand what was going on, not that they could build a model to compete with OpenAI tomorrow. EE (especifically DSP and communications) uses a lot of the same building blocks as ML so yes it does prepare you.

You are right in that the emergent abilities come from humongous amounts of training data and compute which only a handful of companies (and the people working in them) deal with today. I'm sure there's a lot of secret gotchas which only arise at a certain scale so one cannot self-study for them. Companies also guard these wherever possible because of economic incentives. But that applies to most other engineering disciplines. You can take a course in civil engineering and have enough knowledge to reason about why a skyscraper will not fall apart in the wind; doesn't mean you have the ability to build one.

Plausible deniability - The line of attack is in their sessions/prompts data. Just make the prompt pointed enough that the search space is tractable and use your ginormous compute.

> "Of course we don’t know whether that is true"

Yep. Who is verifying these claims? We all know how trustworthy Altman & Co are.

Yeah, I think you can't just throw money randomly at problems and expect results unless you know a line of attack that can get you all the way. OpenAI chose the line of attack only after it became known to them via rumors. They "front-ran" the researchers.

Exactly! This is the real Occam's Razor explanation.

Are you saying there is no search space intractable to LLMs? That wouldn't be possible. AIs are statistical pattern-matchers on steroids. The prompt is key to getting anything useful out of them. They are incredibly useful and major game changers but ultimately that does not alter this fact. People (including OAI) have already tried to solve Millenium Problems with it. That OAI woke up last week and suddenly decided that throwing their researchers armed with millions of compute on one particular idea to a problem is highly suspicious in itself.

Even if OAI had zero data from Buckmaster's sessions, this is in very poor taste and highly unethical. You are front running a researcher just to be able to say you did it first? Tao is right - OAI is treating math results like oil. This is the like Exxon getting a whiff of a massive oil field and racing to the punch by deploying their full crew.

OAI started working on this only after they found out it was close to being solved. They threw a team of researchers who spent sleepless nights + a ton of compute. This is not exactly healthy academic competition - it's like if you spend a year hunting for oil fields and finally find a very promising area to be explored, only to find that Exxon tapped their entire exploration unit to go all in and and find it overnight just to stake claim to the discovery. Tao said it right - math should not be treated as a non-renewable resource to be mined.

OAI doesn't need to mention Buckmaster's name directly in a prompt. They just need to select a basket of sessions that is guaranteed to contain Buckmaster's and then direct the LLM to attack only a specific method/angle. This is trivial to do while maintaining plausible deniability about not using his work.

I'd think nothing is "safe". Anything you say can and will be used by the LLM if it has enough statistical similarity to the prompt. Call it "Ma Random Rights"

Research equivalent of front-running.

I don't think OAI should be given the benefit of doubt. They are doing the research equivalent of front-running. Knowing where to look is one of the main challenges in research. Tristan's argument from his essay was that it is hard to brute force with a vanilla prompt (even for seasoned mathematicians) unless you knew very specifically what to mention i.e the search space would have been intractable even for OAI's compute budget.

"deidentified data" isn't much to go by. Say I prompted the internal model this way - "Hey there's a solution to a unsolved problem X. The solution uses a less known Method Y so don't bother wasting time with the usual methods. Take papers A, B and C as references. Oh btw, here's the last year's worth of data of all prompt sessions that mention this problem. Pay special attention to the ones that mention Method Y and sub-keywords Z,W".

This is obviously all speculation but the timing is very suspect. If OAI actually did this (and I suspect whatever they did is pretty much close to this), I think it is highly unethical.

Try to increase the complexity of the work you're doing. It's not that AIs don't makes mistakes but they are able to pattern match to larger and larger pieces of the problem as more people use it and more iterations of the model are released. So your challenge is now in steering it in a way that these errors are minimised while reducing bloat.

Try to think of yourself as a professor who's trying to come up with a problem statement worth solving. The AI is your "lab" that will help you run experiments.