didroe

It blows my mind further that the presidental pardon exists, and he will face zero consequences for any of this.

I think there's a question of semantics here. RL is way more expensive in terms of compute, as you have to generate a significant amount of output before you can judge whether it's good or bad. So compute is one angle, but another is how much impact RL has on the weights vs pre-training

I think you're conflating prediction and training. The models are auto-regressive so they always predict the next token.

I think it's more likely the millions of dollars of compute than the model, in many cases.

They no doubt have more expensive/powerful models internally, but smaller models seem to catch up fast. So I'm not sure it's about capabilities, but more the willingness and budget to conduct a huge search.

Obviously the more intelligent the model, the smaller/more directed the search is. But they spoke about huge numbers of agents working on Navier-Stokes for example (I think it cost >$10m).

Does every mathematician get cited in every maths paper? I think it's pretty clear who should be cited.

All the methods and data are not public. We don't know what unpublished methods they're using. You can get most of the pre-training data publicly but they've probably spent a ton of money curating it and are now doing things like buying rare books. The RL training data is all (/mostly) proprietary though, and that's the real secret sauce part.

I think a big issue is that the companies pushing AI want it to replace people entirely. So the software around it is designed with that mindset.

I'd really like something that works more like pair programming. Where you share an editor session with the AI, and it's much more interactive. eg. It says "I'm thinking about doing X here, what do you think?". Then you give a response and it continues. Or you can see it doing something silly and immediately stop it and correct something either with a prompt or by manually editing yourself, before letting it rip again. Or just ask it "why are you doing it that way?". Maybe with some control over the speed it's going, for when you feel confident about what it's doing.

Claude has made some positive changes over time where it asks you more when there are different approaches it can take. But I feel like I'm not being brought along with my mental model as much as I would like. A lot of the time it spits out a whole pile of code and then I have to go and build up the mental model after the fact, and correct a lot of what it's done.

The current approach is good a lot of the time though, when you're not really changing anything architectural and just want it to bash out code while you do somehthing else.

It's the exact same training process for both of your examples. I don't really see how you can claim books are not replicated, but that output from other LLMs is.

They have no choice but to train their own model to try and survive. They're paying API pricing for the top tier models but competing against subsidized subscriptions.

The problem is, what's ambiguous or precise is subjective. Your devil's advocate needs to reflect all of the possible readers, and that isn't possible.

There's a good reason we use jargon in professions, or more constrained and less ambiguous languages for maths/coding

The problem is that evolution works on a much longer timescale than the pace of change to the environment that humans cause.

>With RL, models no longer just learn what sounds correct based on patterns they've seen. They learn what words to output to be correct. RL is the process of forcing the pre-trained weights to be logically consistent.

How does Reinforcement Learning force the weights to be logically consistent? Isn't it just about training using a coarser/more-fuzzy granularity of fitness?

More generally, is it really solving the task if it's given a large number of attempts and an oracle to say whether it's correct? Humans can answer the questions in one shot and self-check the answer, whereas this is like trial and error with an external expert who tells you to try again.

The underlying feature of FIAT money creation is debt. And debt is a very natural thing (existing before money) that will just manifest in the crypto system instead.

How do you know it's safe to redeploy? If your entire operation may be compromised, how can you trust the code hasn't been modified, that some information the attackers have doesn't present a further threat, or that flaws that allowed the attack aren't still present in your services? It's a large company so likely has a mess of microservices and outsourced development where no-one really understands parts of it. Also, if they get compromised again it would be a PR disaster.

They're probably having to audit everything, invest a lot of effort in additional hardening, and re-architect things to try and minimise the impact of any future attack. And via some bureaucratic organisational structure/outsourcing contract.