That ~119 GB optimization is great. Hadn't seen that yet. That pops a 4x-Spark setup to 476 GB, which gives a lot more headroom for running something like GLM 5.2 at 4-bit, which isn't completely terrible.
I should have prefaced my post - I almost bought four Sparks a couple months ago, but ultimately opted to buy two more RTX 6000 Pro Max-Q's.
It was a painful choice because the two 6000's were more expensive than four Sparks, and ultimately gave me only 384 GB VRAM.
It was even more painful when GLM 5.2 was released, and a 4x Spark setup could run it at a decent quant, but 4x 6000's cannot with any headroom.
But the 6k's absolutely destroy the Sparks on prefill and inference speed. Model intelligence is compressing. The smaller VRAM pool will matter less over time than slower prefill/inference speed.
That is to say, I'm sure we'll end up with <500B parameter models that are Fable-level in the next 8 months or so. Performant quants of those will fit comfortably in 384 GB.
I should have prefaced my post - I almost bought four Sparks a couple months ago, but ultimately opted to buy two more RTX 6000 Pro Max-Q's.
It was a painful choice because the two 6000's were more expensive than four Sparks, and ultimately gave me only 384 GB VRAM.
It was even more painful when GLM 5.2 was released, and a 4x Spark setup could run it at a decent quant, but 4x 6000's cannot with any headroom.
But the 6k's absolutely destroy the Sparks on prefill and inference speed. Model intelligence is compressing. The smaller VRAM pool will matter less over time than slower prefill/inference speed.
That is to say, I'm sure we'll end up with <500B parameter models that are Fable-level in the next 8 months or so. Performant quants of those will fit comfortably in 384 GB.