Feyd
Joined 4 months ago
Karma
7
Submissions
6
Member since
Jun 25
Recent submissions
Comment · 25 days ago
Hi thx for the reply. Capy works on single card also, but yes the optimisations are done for double card and for nvfp4 nvidia format, that's native on blackwell, so this can also be used in newer datacenter gpus. On the right setups it's faster than strata
Comment · 4 months ago
In the last week I have been researching LLM architectures, putting together all the known ways to train AIs and ended up creating something apparently new.
Long story short, the vRAM needed to train my LLM seems to be up to 5x times smaller than a standard AdamW.
I was studying the ternary weights on my own when I asked myself if it was possible to actually train an LLM with this method. Turns out, yes, it is.
I created a POC about the possibility to train a latent-free LLM, significantly saving memory for training, and shrinking the shipped model size for storage/distribution (inference speed/RAM need a dedicated kernel — not built yet.
During tests on a toy scale on babyLM benchmark (up to ≤125M), performance proved to be on par with other autoregressive models. As of today, there is no optimized kernel, so actual vRAM savings potential and speed-up are not completely fulfilled.
This is clearly still a WIP, as I am experimenting with other configurations and scale-ups. If you are curious about this, the repo is open with more technical information.
https://github.com/ValerioDolci/bitbop
Comment · 4 months ago
From yesterday it changed a lot in it's behaviour for the worst