Comment by n4pw01f

Are you running bare metal or docker? I had to go down to 2 bit on Gemma4 E2B model to run on 8gb on a Jetson

Quality and idempotency is great but it’s still not exactly fast… fast enough and works offline

Is this something that you can get running on Debian?