Anunayj
Joined 5 years ago
Joined 5 years ago
I was able to fit the model with decent speeds (30 tokens/seconds) and a 20k token context completely on the GPU.
For summarization, the performance of these models are decent enough. However unfortunately in my use case I felt using Gemini's Free Tier with it's multimodal capabilities and much better quality output made running local LLMs not really worth it as of right now, atleast for consumers.
I've used Moonlight + Nvidia Gamestream with ~40ms RTT and couldn't feel a difference in competitive shooters, so total latency must be pretty low.
Does it have something to do with the bandwidth requirements? (1 stream v/s potentially hundreds)
1. Recursively lookup DNS, so domains will have to be blocked at the registrar level, since DNS is unencrypted, it can be blocked at ISP level as well.
2. Use a protocol alternative to DNS, a good mature example is GNS. It aims to replace DNS, with a built from group up, modernish protocol. Using a DHT and public-key cryptography.
3. There are "block chain" solutions to the whole domain problem, look at Handshake, ENS etc.
And it doesn't expose ADR/carrier phase.
Contact: me [at] anu.ninja