Comment by imjonse

just use torch/vllm/sglang/llama.cpp and the rest of the ecosystem, most have decent support for AMD, it's not that different from CUDA once you get it working (and it is much easier now than it used to)