jakswa

llama.cpp looks to be cloning openai's realtime websocket API with this audio model as its first use case? I guess this model is either the first speech-to-speech of its kind, OR it's the first good-enough one. I'm thinking I need to give this more testing!

  - https://github.com/ggml-org/llama.cpp/pull/31287
  - https://github.com/ggml-org/llama.cpp/pull/31269

I'm testing this out as a dumb scout model alongside Qwen 3.8 Flash Next (IQ3_S, strata). Currently have it labeled as a "dumb but fast scout, findings should be verified" since some initial testing shows useful-but-sometimes-inaccurate.

on a 7900XT (20GB VRAM, ~800GB/s IO) w/ full context:

    - 8500 tok/s prefill/ingestion
    - 212 toks/s decode/generation

no apology needed, I didn't find any corrections to suggest in your comment, lgtm

I think you were meaning to reply to another comment?

I was recently scanning for newer, more-batteries-included options and this looks well-scoped at first bluff. Thanks for spreading it, might end up in my next experiment!

If you can splurge for the 40GB+ memory it needs, this genuinely seems like a usability upgrade to 27B while meeting/exceeding capability. I was able to get this to complete each of the 3 easy deepswe exercises I threw at it (on q2!). 27B really struggles on deepswe on my 7900XTX GPU -- thinks too much, and slows down a good chunk as context builds. More usage needed!

While I appreciate the viewpoint, I do not think the idea resonates personally. If I'm following, your idea is to take a brain submerged in this feedback loop from childhood and task it with the question: "what should you want to do?" Are you expecting new results? Is there some noble, obvious result you held back that you think this brain can come up with? Pick ourselves up by our bootstraps to go where?

I think the population starting to age is an interesting one -- the first one playing more-complex 2D + 3D games from an early age. I'm in this age group. In a world with nothing left to explore, and humans globally connected by commercialism, I think you don't need to be bothered with what capitalism is making of the world. Our brains are hooked on solving imaginary, complex problems. We're all on some weird ship at sea, migrating toward some unknown old-age destination. Probably an MMO.

Bonsai 2 27B · Radeon RX 7900 XTX

  - 89 tokens/sec generation with speculative decoding
  - 81 tokens/sec at 20k context
  - 474 tokens/sec ingestion at 20k — about 42 seconds
  - 10.1 GiB peak VRAM with a 24k context window

ROCm 7.2.3 · PQ2_0 · Qwen Q4 MTP, draft length 2

---- versus ----

Qwen3.8-27B IQ3_S · Radeon RX 7900 XTX

  - 79 tokens/sec generation on a short coding prompt
  - 61 tokens/sec at 60k context
  - 53 tokens/sec at 95k context
  - 558 tokens/sec ingestion at 60k — about 108 seconds
  - 19.9 GiB peak VRAM during coding tests with a 100k context window

Vulkan · GSQ-RCO IQ3_S · MTP, draft length 2 · vision projector loaded

what kinda speeds do you see on 6700 XT? i'm always conflicted on investing time chasing speed-vs-quality tradeoffs. I've got a 7900 XT (about double the IO throughput). I'll probably end up giving it a go when I find time.

dang only for certain nvidia GPUs, had my hopes up

don't see it in my AWS bedrock model list yet, but boy has bedrock mantle been annoying today with the errors/downtimes, with NO status page entries >_<

Oh. The UI screenshot on github is... not actually in the github repo? It's platform/hosted only? There's my first awkward discovery, but makes sense in retrospect.