eso_logic
Joined 10 years ago
Karma
95
Submissions
12
Member since
Oct 12
Recent submissions
Comment · 3 months ago
This is a great datapoint. Someone else brought up that I should be memory checking the GPUs to understand if things are breaking down.
Comment · 3 months ago
Lesson learned for real, and TIL matplotlib is happy to export SVG. Good to know for next time. I upgraded my lightsail instance size in the meantime.
Comment · 3 months ago
I'm trying bruh fuck!
Comment · 3 months ago
Yes! P4 and the other small cards are fantastic. I've been more focused on developing a cooling system around the 2 slot sized cards so I don't have any of these lying around. Are you using P4 for anything outside of LLM work? I'm interested in seeing their image processing capabilities.
Comment · 3 months ago
16 GPUs would require one or more 220V breaker panels, more akin to an EV charger than a computer. You would also quickly run out of PCIe lanes. My goal with this benchmarking is to think about what is the most cost effective way to fill 4U.
Comment · 3 months ago
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0
Comment · 3 months ago
This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length?
Do you have a set of models you'd like me to look at?
Comment · 3 months ago
Thanks! Yeah this is a major consideration. I have looked at power consumption throughout runs in the past (https://esologic.com/gpu-server-benchmark/#gpu-box-benchmark) and found that for many of these enterprise class cards, they're happy to slam right into the max TDP. So, for doing actual work, you'll be living up closer to the rated TDP of the cards. Recording power consumption is easy on nvidia and I'll likely add this to future versions of the benchmarking tool.
Comment · 3 months ago
Wow holy crap this is news to me! I will have to consider picking some of these up for testing, what is it like working with them?