epups

Some important landmarks since GPT4 was first released (not in chronological order):

- Vast cost reduction (>10x)

- Performance parity of several open source models to GPT4, including some with far fewer parameters

- Much better performance, much larger context window in state-of-the-art closed source LLMs (Claude 3.5 Sonnet)

- Multimodality (audio and vision)

- Prototypes for semi-autonomous agents and chain-of-thought architectures showing promising avenues for progress

Greenwald is an odd one, I'll give you that. He seems to see this issue from a libertarian perspective. He would also say that Germany banning Nazi speech is censorship, for example. It's worth nothing that his last professional occupation was being a stooge in a far right show.

This is what the far right in Brazil wants you to believe, but it's not true. It's especially not true in regards to X, as they have repeatedly ignored judicial decisions that are very similar to those by European courts (ie, remove illegal content).

This is partially the reason why we see LLM's "plateauing" in the benchmarks. For the lmsys Arena, for example, LLM's are simply judged on whether the user liked the answer or not. Truth is a secondary part of that process, as are many other things that perhaps humans are not very good at evaluating. There is a limit to the capacity and value of having LLM's chase RLHF as a reward function. As Karpathy says here, we could even argue that it is counter productive to build a system based on human opinion, especially if we want the system to surpass us.

Intel seems 100% tied to its outcomes as a foundry now. If there is any disruption in chip prodution from Taiwan for example, I can see its stock doubling in price.

The graphs seem to indicate their model trades blows with Llama 3.1 405B, which has more than 3x the number of tokens and (presumably) a much bigger compute budget. It's kind of baffling if this is confirmed.

Apparently Llama 3.1 relied on artificial data, would be very curious about the type of data that Mistral uses.

This article is borderline comical. The thin list of his accomplishments focus on increasing revenue ($15B to $70B) and launching or acquiring successful initiatives (Xbox, Skype, Azure). Then the rest is a recollection of his biggest embarassments, like terrible mobile products, misguided Windows strategy and trailing AWS by 7 years.

Reporting on business issues is always muddled by a lack of proper comparisons, along with cherry picking. For example, this article makes the argument that increasing Microsoft's revenue by 4x was very impressive, even though the stock value stagnated. However, when evaluting his tenure as owner of a basketball club, he is declared successful because its value doubled. The problem is that Microsoft was eclipsed compared to its peers at the time - Google, Amazon, etc. -, and likewise the average basketball club doubled in value as well.

It seems that this would compete with Khan Academy for a similar space. Perhaps Karpathy will aim for adults instead?

The author is making the point that Alphafold 3 is not so impressive - it is simply regurgitating its train set, and it's not so good for inference.

I think his central point is fair and interesting. The test train split is apparently legit, as they used structures released before 2021 for training and the rest for testing. However, there was no real check for duplicates, and the success rate might be inflated by a bunch of "me too", low hanging fruit structures that are very slight variations from what we know.

However, I'm not sure I agree with his skepticism. LLMs suffer from the exact same problems - getting it to write a Snake game in any language is trivial, but it is almost certainly regurgitating - , but can be useful as well. I mean, if for various reasons people are publishing very similar structures out there, there's certainly value in speeding up or reducing that work considerably.

It would be simply stupid that we got so much worse at this technique in the past 10 years. You are welcome to post a link to your data source here, you already had two opportunities to do so.

The article makes it sound like it was ultimately a great decision, because now coal use is at historically low levels while renewables have been increasing. What it neglected to mention is that if nuclear had remained there, the share of coal would have directly and proportionally decreased. To achieve a 5% switch from very dirty to clean energy would be a spectacular feat, and the Germans achieved the opposite when closing their still perfectly usable nuclear plants very recently.