spi

I... doubt that's true? I don't know all the details how these recent image generation models work, but last time I checked people still used diffusion models for it. Diffusion pipelines are _slower_ and _more expensive_ than the language model, not cheaper.

"generalist" multi-modal LLM mostly uses images (and videos, sounds, etc.) as input. I'm quite sure they have an output head for it, as well, used during training, but their outputs are probably not as good. They generate images much faster (hence probably cheaper), too: typically a token is 16x16 pixels, so to generate an image of modest 512x512 resolution would be the same as generating a 1024 token length string, that nowadays is pretty fast and cheap (e.g. with Sol it would be about $0.01). But the result won't be great.

Diffusion models instead are trained to generate a "denoising" step of the image. Typically they have some inputs (e.g. the owl sketch in the blog post and an embedding of your textual request), start either from pure noise or from a noisy version of an input image, and then repeatedly slightly modify it in anywhere between 4 and 100 (or more) steps (hyperparameter to be chosen, and also depends if you start from noise or from something non-pure-noise), each time making a version that the model believes to be a bit closer to where you want to get. The underlying model _could_ be similar to the language model, but I think it isn't, because (1) if you want to call it 100 times in 4 seconds it must be smaller and dedicated, and (2) it's trained to do _exactly_ that, i.e. remove 1/100th of the noise, no more, no less. So it must be a dedicated one.

A few years ago people did research on "one shot" methods (earliest were GANs of course, but also some autoregressive model like a multimodal LLM directly), but as far as I know, quality has always stayed well behind diffusion models. It was faster, though, not more expensive.

Sorry came a bit late to this reply. Interesting, well, nobody says it's a monotonic function :-) in the limit of _very_ large batches you of course are worse off, because you take a very large amount of computation before taking a single step, so if you stop after a fixed amount of time your model just didn't have the time to learn properly. So certainly there is a sweet spot somewhere.

I suppose, the real "function" is a bit more complicated because (1) If you put 2x more data through the same GPU with large enough memory, it will take less than 2x the time to compute (but certainly not 1x). (2) At some point, empirically, increasing batch size makes it _worse_ even if you ignore the additional runtime cost (i.e. stop after n gradient update steps, and not x seconds). To my knowledge, the accepted reason for that fact is that a bit of noise helps in regularizing learning, because overly smooth learning curves end up stagnating in local loss minima more easily. In truth, I think nobody exactly understand how deep learning models work :-)

And to your other question - sorry again for the late answer. Yes, `optimizer.zero_grad()` should always be called directly after `optimizer.step()`, therefore with gradient accumulation once every `n` steps (otherwise, you'd be zeroing out the gradients, so just throwing away all the compute you did in previous steps).

Sorry I just opened that file now, and browsed through it very quickly, but my eye fell on the excerpt: ``` However, we did not observe any speedup by increasing the batch size from 65536 to 131072 for the first stage, thus, we restrict the batch size to 65536 for this stage. ``` which I think is more or less my point: increasing batch size essentially always helps, but the speedup reduces the more you push the batch size. Provided that your dataset is large enough, more batch size will always make you run a bit faster without sacrificing accuracy, but the speedup will be less and less as you increase the batch size, until you are anyway maxing out the power of your GPU and you can't see any measurable speedup anymore.

Mmh not really. As OP shows, speed increases with larger batch size, but only initially, until the GPU has high enough utilization; then speed improvements flatten out (although you might get OOM before that and not "really" see the flat part). Using smaller batch size increases _noise_, so quite literally decreases stability. That might be good sometimes: in the limit case, if the batch is as large as your training set, you'll end up in local minima and not be able to get out of it. But this is true for toy datasets like MNIST, here it's an entirely different beast.

With such large corpora as the ones used here, and very noisy ones at that, gradient updates are very noisy and that can harm quality. Or anyway, common lore is that one needs pretty large batch size to have the language model improve steadily.

Thanks, very nice to see these results! Certainly using GPUs with more RAM makes things simpler to scale. Gradient accumulation is as easy as adding a counter for number of steps and an "if counter % gradient_accumulation_steps:` around `optimizer.step()`, so that can also be tried simply on a single GPU / cheaper GPUs. But if you can just use 8xA100 and your pipeline parallizes well, you also get results (almost) 8 times faster, which is certainly nicer to experiment of course!

A separate comment about conclusions about why they are worse than OpenAI GPT2 - which to me feel to be missing the point.

One main point is batch size - I'd agree with Gemini here. Batch size <= 5 with 1024 seq len is really tiny. Nowadays models are trained with effective batch size of millions of tokens in total. Of course, this won't fit into memory, one uses gradient accumulations to that purpose, again as mentioned by Gemini.

Training duration is definitely also a reason - models do get better over time, otherwise people wouldn't train so long wasting millions :-) just how long for optimality is unclear, but certainly < 2 days is not optimal even at this "small" scale.

The optimizer could also play a role. As the author mentions, a fixed learning rate is hardly optimal, it is typically both increased in the beginning ("warm up", but that's for stability, if training works without, that's not an issue) and scaled down at the end ("cool down" - that is, annealing, with cosine as mentioned in the article). This generally squeezes out a bit more performance. Also, while it's true that dropout was used back then (might be useful for many epochs, likely only harmful for < 1 epoch), using _both_ dropout _and_ weight_decay > 0, as the author does, is probably wrong and makes training too slow & careful to get good results. Also, even if used, a "good" implementation of weight decay should skip some layers like embeddings and biases (GPT2 did that, and it's relatively important to do so).

On the other hand, I'm pretty sure that using mixed precision and TF32 has absolutely no downsides. It's really standard nowadays to use either mixed precision (FP16 gradients + FP32 base weights) or directly BF16 ("brain" float 16, a bit like the TF32 described there, but with only 16 bits) and I have almost never seen either one fail... and when it does, it typically fails spectacularly, with NaN losses or the model degenerating to trivial performance.

Aside from the weirdness of calling "good old" something that was released 17 months ago :-D I mean, deep learning is evolving at crazy rhythm, but you just can't assume a good paper gets written in days.

That said, as others have pointed out, and as it's also written on the blog post, they are entirely different methods. QLoRA requires access to the full training data, while theoretically you can apply SpinQuant to any given model. For example, they also apply it to Mistral, not only to their LLaMA.

(QLoRA also takes some time and compute to apply, but since SpinQuant also implies learning some weights, I don't know if it's actually faster/cheaper, too)

I know nothing about what makes an industry succeed or fail, and also nothing about web tech, but working in the field I can comment on:

> tensorflow looks like currently loosing to pytorch - seems like google got bored and more development is for JAX, Keras wrapper

Well, TensorFlow doesn't "look like currently losing", it has already lost since a long time. I haven't seen a decent paper release code in TensorFlow in years, and all the references I see to TF online are job posts from "older" companies (to the point that, if you are looking for a job in data science, seeing TF mentioned in the job post is kind of a red flag of a place you don't want to be).

That said, I am quite certain that this has only a small impact on why Google is losing terrain, and even on why it is behind in AI (which is also debatable: narrative aside, Gemini is not that much lacking behind competitors). Certainly if TensorFlow + TPUs turned out to be better than PyTorch + GPUs they would have had a lead to start from, but if that was so important, Meta or NVIDIA would have created the first LLM, not OpenAI.

Simply, sometimes stuff happens, you can't predict it all.

I know this is HN and here it's not a popular opinion, but maximum security is _not_ always a good idea. Even setting aside the problem of many different actors having to access these details mentioned below, there's value in a simple login process. Specifically for airplane tickets, the most common ones I had to struggle with multiple times are retrieving reservations bought from a different computer, or by a travel agency. In all these situations, it was exactly the simple approach that saved me. If 2FA was mandatory, the best case scenario was that the travel agency would have to send you a separate e-mail with details about how to access their portal where this 2FA would somehow work. The number of systems multiplies, the number of credentials to remember does, as well. If you are not from your usual workplace (and chances are, if you are travelling, you are not) or from a shaky connection (same), you are in a real problem. In a time-critical scenario, which makes it really worse.

Implementing a "secure" connection here would be a sure road for pain ahead, at least it would need the airplane company to increase customer support a lot, and likely a lot of bad publicity every time something fails. Delays cost money, especially in this industry. And what would you get for that? The safety that, if you publish a picture of your reservation / boarding pass online, nobody can log in with your credentials and cancel your flight? That's a rather niche and very targeted risk, which is better handled by a single customer support agent who, simply, issues you a new ticket.

(by the way, by the time you have checked in and your boarding pass has been issued, a lot of companies just don't allow you to cancel anymore, so it's really a non-issue?)

I'm into AI but not into sound, so I might be saying something stupid here, but I think using something like this for very high volume like concerts would be possibly outright impossible, but, even if not, certainly quite dangerous and therefore not commercializable.

My understanding is that to "mute" a sound, you need to inject another wave that is exactly the opposite, with the exact same volume and in perfect sync, so that the two waves interfere destructively. However, in general but especially in AI, you can never guarantee 100% accuracy. If you use this technology to "silence" a background fountain, and something goes wrong, at worst you get a lot of noise that make you grimace and remove them. If at a concert with 100+ dB of music you get an error and your headphones start producing a similarly loud, but not perfectly aligned noise right into your ears, you probably won't have the time to remove them before damaging your hearing system.

In general, I think that having a tool that drives 100+ dB straight into your head is probably not a wise idea :-)

Yep that also sounded weird to me. I had, IIRC, three of my wisdom teeth removed as a teenager, I was living in Italy back then. I think two of them in a single session. General anaesthesia wasn't even an option, the whole thing happened in a normal dentist cabinet with a local anaesthesia to the relevant half of the mouth. I distinctly recall the dentist complaining that for one of the teeth my roots were particularly strongly attached to the bone, and he had to push and lean on it, _hard_; it didn't really feel painful, except that my jaw was aching on the opposite side (the mostly-non-sedated one) due to the pressure he put on it.

In fact, I think people and doctors alike tend to sedate much less in Italy - maybe not completely unjustified from a few things I've read in this thread. Back then, the normal drilling & filling tooth cavities mostly happened without any anaesthesia at all, local or otherwise. Frankly, that was quite painful, whenever the drilling happened to touch a nerve, and I really don't feel like experiencing it again :-) and I think at least this changed since.