zepearl

Hehe - yeah, I was pretty sure that you just overlooked it - pointing that out was irresistible :o)

I know nothing about "vLLM" -> anything better compared to "llama.cpp"?

I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.

Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).

I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.

The parameter "reasoning_effort" is something new, or am I wrong?

  Qwen3.8 comes with official support for reasoning_effort, which can be used to adjust reasoning depth and control cost:
    - xhigh (default): for complex tasks demanding thorough analysis
    - medium: balancing accuracy and speed
    - low: efficient reasoning optimizing for speed and cost
  In addition, preserve_thinking is enabled by default for all workloads for the best out-of-the-box experience.

Asking because in my case (OCR of scanned historical "National Geographic" magazines) the LLM trying to merge text split into separate columns was running in circles from time to time and needed a lot of prompt tuning when using Qwen 3.0/3.5/3.6 (still needs from time to time).

Gentoorian here as well, cheers.

I use it since ~20 years on most of my notebooks/PCs/root servers mainly because of its flexibility and I often end up wanting to use the very latest versions of software (e.g. ZFS, kernels, etc...) without struggling as most things are available from the official central package repository. I admit that updating/upgrading the OS is a pain from the time perspective (because of the long compilation times). As a programmer/developer the experience is as well fantastic as most of the libraries, compilers, interpreters, etc are very well integrated or easily installable.

I did try to use a few times Linux Arch, but I never fully liked its AUR.

For VMs I basically use always either Debian or Linux Mint with Xfce.

For gaming, mediacenter-PC and webradio-miniPC I use Linux Mint with Xfce.

It's funny that whenever I need assistance with some task (fixing a problem, doing something weird, ...) and I mention to Google Gemini something like "I'm using Gentoo Linux blahblah" then often Gemini starts its reply with something like "Ah, a picky Gentoorian, I see - then let's do this the complicated way..." :o)

> I have trouble getting my computers to copy data between each other anywhere near the theoretical limit for a 1 Gbps NIC, despite being connected through only a single switch.

Maybe your LAN cables have deteriorated?

Reaching 100 MiB/s should be easy peasy for a PC; in my case (2 10Gbps switches) by using nvme SSDs I reach almost 950 MiB/s and with 4x raidz1 HDDs I reach 450-650 MiB/s.

Until a few years ago I was smoking in my flat and often I had to change lan cables (whatever's in the smoke was sticking to the connectors)

In my case it somehow depends a lot on the task being performed... .

E.g. when doing text transcription/OCR from images (Qwen 3.6 27B Q4_K_M by Bartowski) with a context size of ~50k I get a pp of ~460 tokens per second and a generation ranging from 35 to 45 tokens per second (using "--spec-type draft-mtp --spec-draft-n-max 2" currently with llama.cpp b6548).

On the other hand when handling code (Qwen 3.6 27B Q5_K_M by Bartowski) with a context size of 128k I get a pp ranging between 500 to 1500 tokens per second and a generation between 25 and 40 tokens per second (using in this case as well "--spec-type draft-mtp --spec-draft-n-max 2" currently with llama.cpp b6548).

Anyway in theory with "--split-mode layer" I think that it's anyway the slowest card that drives the overall performance (I do see in "nvtop" that usually the 5070 is ~25% active, the 5060 ~50% and the 3060 ~75%).