pi-victor
Joined 8 years ago
Joined 8 years ago
This takes me back to the first commercial software i ever worked on around 19 years ago, it was an accounting software written in Visual FoxPro. It's a super easy language to get started with programming and the fact that you can drag your widgets on the screen helps a lot. I remember using hungarian notations too. In hindsight that's both good and bad, because the moment you move to another language that doesn't offer an IDE where you can drag widgets with your mouse on the screen to position them and you have to work with containers, programmatically, you quickly realize how little you really know about creating UIs with forms. I don't think i had an opinion back then if VFP was good or bad but I remember the IDE would crash when you tried to build your project sometimes with a pop-up that would say "Catastrophic failure!". I always thought that was super funny and to this day i still have the screenshot. I think about a year or so before they killed it, they releases SP2 which was a bit more stable.
my setup
# Logical CUDA0 = RTX 4090, logical CUDA1 = RTX 3070 export CUDA_VISIBLE_DEVICES=0,1
cd ~/projects/misc/llama.cpp/
exec ./build/bin/llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M --mmproj /xx/xx/xx/xx/xx/mmproj-Qwen3.8-27B-Q8_0.gguf --host 0.0.0.0 --port 8080 --jinja --parallel 1 --split-mode layer --tensor-split 6,1 --fit on -fa on -c 98304 -ctk q8_0 -ctv q8_0 --image-min-tokens 1024
i load more on the 4090 because it's faster.
usually the temp stays around 65 for both. utilization for 4090: 70-90% 3070: 30-50%. I get around 30-40 tk/s. if i offload more to the 4090 the tk/s goes up, but i stress the card too much and that thing now is worth its weight in gold.
note: the pi-llama plugin needs a patch for pi to send the model vision capabilities, seems it doesn't work out of the box.
make this available on testflight, this looks great, would love to daily drive it.
+1 for the accessibility features, great job!
thinking about it, it would be funny to first run alibaba's tool and then run parley after.
posted it here a few days ago: https://news.ycombinator.com/item?id=48369782 i guess with AI there are too many Show HN now, and i never got any type of feedback.