joshhart

Google stock dipped because search traffic is shifting to LLMs. Even if Gemini got a large percentage of that, margins are much worse.

An impressive amount of gall. Meta will only pay the full settlement if their competitors ALSO pay a fine. From the article:

The remaining 30% (approximately $5.3 billion) will be released only after two specific conditions are met.

YouTube and TikTok implement a one-hour Daily Limit, Night Mode, and age assurance measures. YouTube and TikTok each pay an amount matching the 30% figure, with half of the remaining funds tied to YouTube's payment and half tied to TikTok's.

Discriminative has a meaning in machine learning that I think is relevant here. There are "generative" models like LLMs that are learning joint probabilities P(X, Y) and "discriminative" models like logistic regression that learn conditional probabilities P(Y | X)

According to the article, the major effect is they are catching and prosecuting cheaters with a secondary effect that students are over-relying on them for homework or practice, and are unprepared for exams.

The article also talks about the fact that the SAT was banned. I really am curious what percentage of the effect is from no SAT vs LLMs. I guess they haven't had the SAT for a while though so it's gotta be the LLMs?

If you have good ideas that have a nice return on investment and leverage existing skills, sure. If you don’t have good opportunity laying around, best for the business to switch to maintenance mode, which means cutting staff. Or maybe cut staff, then use equity to buy growth via acquisition. It really depends on the business. Block’s growth has slowed so perhaps this would have happened anyway and AI is just what’s getting the blame.

I thought this wasn't viable due to cooling requirements - how do you cool massive amounts of compute when the only option is to radiate it into space - nothing to convect it with?

Also, the incredible amount of grift here with the left hand paying the right is scarcely believable. Same story as Tesla buying Solarcity. Board of directors should be ashamed IMO.

Huh. The standard in your case is to measure waist circumference if BMI is high. Did no doctor do that? As long as you are below 40” or 37” if Asian you are considered good to go.

This is super awesome, but how in the world did they come up with a name "Nano Banana Pro"? It sounds like an April Fools joke.

I believe it's intended to convince the audience they are experts, that this type of thing is dangerous to a business, and they are the ones doing the most to prevent it. There is no explicit statement to this effect, but I get the sense they are saying that other vendors, and especially open models that haven't done the work to curate the data as much, are vulnerable to attacks that might hurt your business.

Also a recruiting and branding effort.

All of this is educated guesses, but that's my feeling. I do think the post could have been clearer about describing the practical dangers of poisoning. Is it to spew misinformation? Is it to cause a corporate LLM powered application to leak data it shouldn't? Not really sure here.

So the inference speed at low to medium usage is memory bandwidth bound, not compute bound. By “forecasting” into the future you do not increase the memory bandwidth pressure much but you use more compute. The compute is checking each potential token in parallel for several tokens forward. That compute is essentially free though because it’s not the limiting resource. Hope this makes sense, tried to keep it simple.

A single node with GPUs has a lot of FLOPs and very high memory bandwidth. When only processing a few requests at a time, the GPUs are mostly waiting on the model weights to stream from the GPU ram to the processing units. When batching requests together, they can stream a group of weights and score many requests in parallel with that group of weights. That allows them to have great efficiency.

Some of the other main tricks - compress the model to 8 bit floating point formats or even lower. This reduces the amount of data that has to stream to the compute unit, also newer GPUs can do math in 8-bit or 4-bit floating point. Mixture of expert models are another trick where for a given token, a router in the model decides which subset of the parameters are used so not all weights have to be streamed. Another one is speculative decoding, which uses a smaller model to generate many possible tokens in the future and, in parallel, checks whether some of those matched what the full model would have produced.

Add all of these up and you get efficiency! Source - was director of the inference team at Databricks

This is pretty exciting. Now an organization could produce an open weights mixture of experts model that has 8-15b active parameters but could still be 500b+ parameters and it could be run locally with INT4 quantization with very fast performance. DeepSeek R1 is a similar model but over 30b active parameters which makes it a little slow.

I do not have a good sense of how well quality scales with narrow MoEs but even if we get something like Llama 3.3 70b in quality at only 8b active parameters people could do a ton locally.