jborak

I already had 2x 5070's that I had purchased a year or more ago, so getting an additional two to fill up the PCIe slots on the motherboard seemed reasonable.

I did some math/shopping as well. To get 48GB of VRAM you can get 2x 3090s but that is $3k. A single 5090 is $4k but has 32GB, great for running models like Qwen 27B but maybe nothing else depending on your model settings. Already having 2x 5070, where each card is around $600, it made sense for me to get two more which was $1200 and the memory speeds aligned.

The best value option if you're building from scratch is go with 5060 ti (16GB VRAM). Each of those cards are $570/each on Amazon, cheaper than 4x 5070's. Only downside is memory speed is slightly slower, but you wind up with 64GB of VRAM and you can run big models and small models alongside each other comfortably.

In my setup I ran Qwen3.5 9B for fast inference on simple things and Qwen3.6 27B Q6 for coding work. But I ran into stability issues, so I use llama-swap to dynamically swap models. But with 64GB of VRAM, you wouldn't have that issue. There is overhead to loading LLMs into VRAM that isn't clear, so having extra VRAM is a helpful buffer.

I'm using 4x RTX 5070's and first-gen AMD threadripper (1950X) to run Qwen3.6 27B (MTP) Q6_K with llama.cpp and it works great as a daily driver with Pi. Around 50-60 toks/sec. I also connect a few other applications to it such as OpenWeb UI and recently set up Bifrost, an LLM gateway, to be the primary access point for the models I serve.

I've tried other models such as Qwen3.6 35B A3B and I've found that 27B works better for me when it comes to coding. It's slower being a dense model but the quality seems much better. Inference on my system for Qwen3.6 35B A3B is around 130-140 toks/sec, non-MTP, which is insanely fast!

You don't need 4x 5070's to run Qwen3.6 27B, three or maybe even two will work. However, I use MTP (multi-token prediction) to speed up 27B and that eats up more memory because the draft model requires its own context.

Another thing to keep in mind is that the tools you're using have their system prompts that are loaded into the model for each conversation. When I fire up Pi, working with the model is very snappy at start. When I interact with the LLM via Hermes CLI, it's much slower. That's because each prompt with Hermes is loading so much stuff (skills, tools, etc.) into the context and then it's there forever until the conversation ends.

I like running models at home for privacy, but I also like how there are no quotas, usage isn't a worry. If the future is "loop engineering" then you will be burning through tokens and $$$ using a cloud models.

My system idles around 200W and is around 350-450W when inference load is high. Decoding (token generation) isn't all that efficient, and your GPUs sit idle more than you think during inference. Advancements like diffusion may 1) speed up decoding and 2) let you utilize more of your idle GPU.

Our services were used for C2 as well. I investigated it a bit but eventually decided to just drop TCP forwarding from our free-tier and that reduced our abuse/malware reports for C2 over TCP to zero essentially.

One path I looked at was to use the VirusTotal API to help identify C2's that other security organizations were identifying and leverage that to automatically take down malicious TCP endpoints. I wrote some POCs but did not deploy them. It's something I plan on taking up again at some point next year.

Thanks for sharing this. I run packetriot.com, another tunneling service and I ended up writing my own scanner for endpoints using keyword lists I gathered from various infosec resources.

I had done some account filtering for origins coming out of Tor, VPN networks, data centers, etc. but I recently dropped those and added an portal page for free accounts, similar to what ngrok does.

It was very effective at preventing abuse. I also added mechanism for reporting abuse on the safety page that's presented.

I use ProtonMail and overall it works pretty good and the value is great. The product is focused on security first so I think user experience suffers a bit. It's works fine but if you previously used a mail client like Spark on your phone you'll definitely miss it.

The mobile app is okay, it's good enough for reading and sending short replies. The web interface is alright. I like to use the bridge application from ProtonMail on my desktop and use a mail client like Thunderbird or Apple Mail.

It's great to have options and alternatives. I feel like calling your application a copy-cat is being harsh. What you've discovered is that there is a now a market of providers for the category of your application.

If there was only one instance of any sort of application or tool the world would be boring and we'd be stuck with only stale options.

A lot of people have seen the Clint Eastwood "Man with no name" westerns but if you haven't watched "Once Upon a Time in the West" I would definitely check it out. One of the best movies I've seen and if you're not fan of westerns (which I wasn't) you will appreciate the genre some more after seeing this movie.

It's built exactly for that. The tunnel is established in reverse over a websocket, so outgoing traffic from your PC is TCP:443 which looks like normal TLS traffic.

I've been using this for almost 6 months and haven't seen any throttling. Your Netflix, YouTube or any other streaming traffic will dwarf what bandwidth you consume, unless we're talking terabytes...

Hey HN, this is the author, thanks for checking out my project and any feedback. As I built this I found alternatives, here are some differentiators:

* access logs & metrics - these types of systems obscure this info (src IPs, etc...)

* system health checks - probes if the destination (upstream service) is available, if not, we'll email you

* self-hosted - want to host the edge server software on your own, you can do that

* specification - the plan is to publish the protocol and spec so others can write clients or integrations

Glad to help with questions, cheers.