idonotknowwhy
Joined 9 years ago
Karma
619
Submissions
402
Member since
Jan 31
Recent submissions
Comment · 2 months ago
I just looked through the slop Readme file, it looks like teams is supported.
Comment · 2 months ago
Fortunately llama.cpp works well with 4 concurrent requests now. This is the default, and you can increase or decrease it with -np N
Comment · 2 months ago
If it's a Nvidia card 3000 series or newer, I'd try 4.0bpw ExllamaV3 if you haven't already. Otherwise it look like UD3.0 Q3_K_XL based on the Unsloth blog post.
Comment · 2 months ago
>But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL
For those of us with a 16GB GPU, how do they compare with ExllamaV4 at 4-bit (4.0bpw)?
It looks like that fits in 12.5GB of VRAM since embedding are left in DRAM, Unsloth Studio and other llama.cpp derivatives have to load these weights in VRAM for tied embedding models like Qwen3.8.
ExllamaV3 4.0bpw fits in 12.5G of VRAM and beats IQ4_XS according to the measurements here: [turboderp/Qwen3.8-27B-exl3](https://huggingface.co/turboderp/Qwen3.8-27B-exl3)
But those were compared against UD2.0 I guess. Also plans to support these (SOTA) quants in Unsloth Studio?
Comment · 3 months ago
How are you not blocked by banking, shopping, even sometimes google search?
Comment · 3 months ago
Server side tracking.
Comment · 3 months ago
Bots wouldn't miss the middle paragraph.
Comment · 3 months ago
That's amazing. I didn't even realize but it seems I read the first paragraph and the last sentence, concluded "moron" and scrolled down.
It's only because of your comment that I re-read the their post.
Comment · 3 months ago
Then the creator should have a sign up button, not a fake chatbox.
This dark pattern is reminiscent of those online test sites in the 2000's where you spend 10 minutes filling out some quiz, then get prompted for an email address to see the results.
https://chat.mistral.ai/chat <- let me chat and actually responded without signing up.
MoonshotAI had this fake chat box dark pattern.
So I signed up with Mistral instead.
Comment · 3 months ago
CC actually prompts Claude about this by default in the ~20k system prompt and instructs it to avoid 300s timeout and to be mindful of the 300s cache expiration.
Comment · 3 months ago
>the only open model that can do tasks well and efficiently is Qwen3.7-Max
Qwen3.7-Max is a proprietary, closed weight model.
Comment · 4 months ago
2 reasons.
First, it's not really "1 bit", actually much closer to 2-bit. IQ1_M is actually 1.75bit and IQ2_XXS is 2.06bit This is from the ./llama-quantize --help with most of the quant types and their size in bpw: https://pastebin.com/bCUqGfeE
And to elaborate on the "dynamic" aspect inconito said in the other comment, if you click on one of the .gguf files in huggingface:
https://huggingface.co/unsloth/GLM-5.2-GGUF/blob/main/UD-IQ1...
There are a lot of Q5_K, Q6_K, etc tensors. Only the routed experts (ffn_gate_exps.weight, ffn_up_exps.weight, ffn_down_exps.weight) are heavily quantized, and it looks like the down_proj is actually iq3_xxs for this model.
Comment · 5 months ago
Yeah, be sure to put everything in tables and include “best balance” for a mediocre option and “great value” for any completely useless options.
Also make sure the shape of the paragraphs is completely uniform.
Comment · 5 months ago
Am I the only one who doesn't get angry at LLMs?
From the blog:
>I don’t really get anything useful out of these postmortems (e.g., clues about how to rephrase my instructions)
Unfortunately, an LLM can't actually reflect or advise how you could have improve the prompt. Otherwise we could give them a sample output and say "Generate the prompt that would produce this output.
Comment · 6 months ago
How did you do this?
I tried to get notepad and mspaint from an older Windows 10 build -> Windows 11 on a surface pro, but gave up after a few hours...