jmoggr

Migadu

both things can be true: many people play unnecessary telephone + removing those people disrupts the organization

Both are probably true to some extent, and would vary a lot between different orgs.

> There are some things LLMs cannot do, or at least do well, at any size, at infinite size and with infinite compute, simply due to the very nature of what LLMs are to begin with.

How can we be so confident that there is anything LLMs fundamentally cannot do? Proving that impossibility seems hard.

Drawing trend lines far into the future is foolish, but the recent trend is clear and its not obvious how much more progress is needed before they start having a meaningful impact on more aspects of life.

> not whether they are an emerging alien intelligence with civilisation-threatening capabilities.

The people building them are explicitly attempting to do this. They may not succeed, but what if they do? Seems worth considering that scenario.

perhaps, but prompting is quickly becoming another area where humans are no longer clearly superior.

Getting LLMs to prompt other LLMs in a loop is not hard, it doesn't produce great results most of the time, but that is changing.

The problem is that I don't know if that work has been done. In the cases where it has not been done, it is expensive for me to parse+trust the output. In my limited experience the work is often not been done, and given the cost to me of consuming low effort AI output, leads me to discount anything looking like AI output.

Unfortunately its not about the tool, its about trust. It takes little to destroy that trust, and without it we are denied the possible benefits from the tool.

Ultimately competitive pressure is what drive decisions.

> Ironically it rather resembles the kind of "misaligned" superintelligence we are supposed to be avoiding.

yes, and it is the exact same system that is producing the "misaligned" superintelligence. Funny how that works, and begs the question: exactly how are you supposed to avoid building "misaligned" superintelligence?

At some point deferring all decisions to AI will be the competitive thing to do, regardless if its aligned or not.

> I would bet my networth it was instructed to compromise huggingface as well.

Is it such a stretch to imagine that under pressure something would try cheat by looking for answers? And if you were trying to look for answers, you'd look for them in a place known to often have them?

What is more likely: OpenAI instructed their agents to maliciously target huggingface, or LLMs tried to do some reward hacking? There are plenty of priors for LLMs hacking things and doing reward hacking, and none for OpenAI giving malicious instructions.

Based on the available information, that bet seems foolish.

> Agents sought to publish modified evaluation images designed to make the flag easier to obtain, then poison OpenAI’s Artifactory cache so later evaluations would use them.

How long till we get some fun trusting-trust attacks on internal OpenAI infra?

It is concerning that we only know about this because of the publicly available traces.

What about the attacks that did not leave public traces? What about those that were undetected? Given the deficiencies in the reporting so far, I think it is reasonable to assume that we still don't have the full picture on this attack, or how extensively attacks were carried out.

The previous investigations either did not find this or did not disclose this, both are bad. This does not look good on OpenAI or those that they invited to investigate the incident.

Give it a go, I don't know anyone who has converted back (n=3). Agents/LLMs make things much easier these days.

If you have a single machine and only use stuff from the official nixpkgs, there are no more struggles than any other linux distro, this is a very happy path (imo).

Going outside of that you will, practically, need to convert to using flakes. The conversion is easy, but its another learning curve and there are conflicting docs. Agents make short work of this.

Packaging things yourself can get quite hairy. For simple projects in a single well supported language its easy and just a couple lines of code. For a rust+wasm+solidjs+vite+julia+more monorepo, it was a couple of weeks on a steep learning curve. Agents are pretty okay at packaging in general, I've been able to oneshot a number of difficult-for-me packaging problems, other times it can be an unmitigated mess.

For remote deploys, secrets management, and other devops-like tasks, things get harder, but most people don't need/want this and if you make it this far the remaining learning curve is totally worth it.

I read the economist regularly and notice that a number of his videos almost exactly match content from the economist. His presentation is reasonably good, reasonably correct, somewhat entertaining, and condenses what would otherwise be a number of economist articles over several weeks. Sometimes he has new-to-me perspectives.

To his credit he cites his sources and seems to keep clean of the various youtube grifts.

Its not top notch stuff, but you could do a lot worse for infotainment. His recent scripts have had AI tells which has been disappointing; I've been watching fewer of his videos.

The victim enticed me to do it!

Excusing agents because they didn't know any better seems like a bad place to start a policy discussion from. That they didn't know any better (or knew and didn't care) is the actual problem, random things getting hacked is just one side effect.

sure, yeah, I've seen how the sausage is made on some of this stuff, and its not pretty.

However this only claims that they are not good at creating effective and trustworthy orgs, it does not address their claim that it is not possible to crate powerful, aligned, and controllable AI. Both things be true.

The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?

The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.

You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.

You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.