cpldcpu

I mean, this paper clearly shows that all models severely degrade for low bit quantization. Also, these experiments were performed on relatively old models that are not trained as close to capacity as newer ones.

The capacity of your model scales with total number of bits in your weights. Shannon entropy.

Yes, there are many way to distribute the error. And LLM are usually not trained to capacity or are prevented from this due to information bottlenecks in the gradiant flow. This allows unlocking wasted capacity by distribtion of the quantization error. But I am pretty certain that there is no way to trick Shannon.

I understand the obsession with low bit quantization, but it is empirically quite evident that it is not possible to compress models to less than 4 bit per weight without severe loss of capabilities².

It may be nice as an experiment, but it is obviously a very inefficient route for model training: spending all the flops on a saturated model only to prune its capabilties.

²As to why, I have seen few explanations. But the empirical evidence is there.

You mean the one they copied from Microsofts paper? (properly cited as well)

Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?

Well, that is what video models do.

But keep in mind: So far there is no way to train the models while completely avoiding memorization and only including generalization. That would be a great way to avoid any copyright issues, but all attempts I have seen so far were fairly limited.

its doing video, audio, images and motion. I think that counts as multimodal.

>When Intel finally shipped it at the 45-nanometer node in 2007, Gordon Moore called it the biggest change in transistor technology since the late 1960s. The breakthrough was not the material. It was learning how to process the material at scale.

Its curious that they picked this example. The challenge with HKMG was not the material itself, but how to integrate into into the transistor stack.

There were two completely different approaches: Gate first and replacement gate. Gate first is what the industry was already using for silicon oxide so everybody tried to go with as little change as possible. Only intel decided for replacement gate, which worked much better and reaped some other benefits on the way.

This was a watershed moment in the industry and ultimately led to some of the players dropping out of the cmos race.

But is this really a "scale-up" problem? It required development of novel manufacturing processes (atomic layer deposition), but was still mainly a process integration and device engineering topic.

The part of the thesis I have to agree with is that there is a data problem. The development above relies on executing lots of time consuming and tedious split experiments that often cannot be parallelized. The outcome of this relies heavily on the experience and diligence of the experimenters.

It's probably well suited for an "autoresearch" approach, bridging to the phyiscal world and dealing with the timescale is the challenge.

hm.. has been quite a while for me. The good thing about the Tang Nano is that it is supported by the Yosys open source toolchain. There are quite a few resources on the web when you search for the combination.