I need to play around with this.
I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.
I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.
I must be doing something wrong.
Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".
llama-server \
-m ~/models/tiel-coder/Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf \
-ngl 20 \
-c 16384 \
--jinja \
--alias tiel-coder
using it in opencode
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"llamacpp": {
"npm": "@ai-sdk/openai-compatible",
"name": "Local llama.cpp",
"options": {
"baseURL": "http://127.0.0.1:8080/v1"
},
"models": {
"tiel-coder": {
"name": "Tiel-Coder 35B A3B",
"limit": {
"context": 16384,
"output": 4096
}
}
}
}
},
"model": "llamacpp/tiel-coder"
}
I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.
I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.
I must be doing something wrong.
Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".
llama-server \ -m ~/models/tiel-coder/Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf \ -ngl 20 \ -c 16384 \ --jinja \ --alias tiel-coder
using it in opencode
{ "$schema": "https://opencode.ai/config.json", "provider": { "llamacpp": { "npm": "@ai-sdk/openai-compatible", "name": "Local llama.cpp", "options": { "baseURL": "http://127.0.0.1:8080/v1" }, "models": { "tiel-coder": { "name": "Tiel-Coder 35B A3B", "limit": { "context": 16384, "output": 4096 } } } } }, "model": "llamacpp/tiel-coder" }