Comment by AntiRush

I've been working on support for this model in ds4 on the RTX 6000 pro - it's been really great for my use cases. The ds4 q4 quant performs a lot better than other similar sizes that I've seen.

Using the Q4 quant on an RTX 6000 Pro Workstation Edition at 450 watts:

  Code: prefill 1,251 tok/s decode 255.26 tok/s
  Prose: prefill 1,251 tok/s decode 198.78 tok/s 

Most important for me, I can run 4 concurrent streams at 400+ tok/s.

https://github.com/fairfieldt/ds4