Comment by nbardy

Everyone keeps repeating this who doesn’t understand the underlying technology.

Small llms are still way more efficiently server on big GPUs.

Sharing server capacity takes advantage of the massive parallel throughput and sharing of memory bandwidth.

You are sharing the GPUs with thousands of concurrent users.