nzeid
Joined 2 years ago
Joined 2 years ago
ROCm promises a 30-50% prompt processing speedup. This is REALLY important for my workflow so I've been trying to get this shit to work for months. But no release before v10 worked well enough with any engine for it to matter.
The llama.cpp release binaries for ROCm (10) FINALLY work on gfx1501 and its relatives (with the correct shell variables), but the prompt processing boost doesn't materialize and the token generation speed decreases.
There continues to be a chronic problem across all engines with the ROCm integration for UMA devices. The good news is that some improvements have been made to that end for Vulkan, so more recent llama.cpp Vulkan binaries are now faster.
This framing is confusing because OPcache has for the past 15 years taken care of exactly this bootstrapping you're worried about. In a competing share-nothing setup you merely come in at a tie for performance (barring first run of `require`). I do not see 10-50ms latencies in any of my deploys which is why I'm skeptical.
That said, the big sell of your project for me might be memory management - less memory means more workers which is always a good thing if it comes at no cost to latency.
What is the setup/benchmark that caused you to see this?
You're probably asking if they're able contribute anymore?
This is a very easy sell for me because everything these days is doing the infuriating gray text on gray background.
> Neobrutalism is a mix of regular brutalism in web design and more modern typography, illustration, and animation standards. Neobrutalism refuses the usual components of UX-UI design and embraces uncomfortable design elements, and it is more fearless to use distinctive color palettes.
Bad managers hollow out companies. It's quite devastating.