Comment Comment by underlines - how much vRAM needed to run each model parameter size?- any inference optimization we can use similar to StableDiffusion, to bring down the vRAM requirements?I only know about these:- use 8bit precision- https://github.com/bigscience-workshop/petals- https://github.com/FMInference/FlexGen- https://github.com/microsoft/DeepSpeedAnything that could bring this to a 10GB 3080 or 24GB 3090 without 60s/it per token? parent underlines 3 years ago
- any inference optimization we can use similar to StableDiffusion, to bring down the vRAM requirements?
I only know about these:
- use 8bit precision
- https://github.com/bigscience-workshop/petals
- https://github.com/FMInference/FlexGen
- https://github.com/microsoft/DeepSpeed
Anything that could bring this to a 10GB 3080 or 24GB 3090 without 60s/it per token?