therealpygon
Joined 3 years ago
Joined 3 years ago
That makes the original statement even less accurate.
The lack of AI mental model between conversations or even within the conversation is a definite frustration. “No, I didn’t just decide to entirely change everything I’ve planned based on one ambiguous word…”
Your use of a separate session for q&a is actually something I find interesting in the new Granite Switch architecture’s KV Cache Normalization, where the kv cache is essentially frozen while the model performs certain other tasks in order to prevent the mental ‘conversation’ from getting polluted. However, I’m also assuming what you mean by “helps grounding” is that it helps keep the agent grounded in the context by not introducing a bunch of external information, which is certainly beneficial.
Outside of major players like Amazon/Google which have compatible offerings available, I believe together.ai, fireworks.ai, and deepinfra.com are, as best as I can tell, US-based (if that matters in your situation) SOC2/HIPPA/ZDR. Each of them offer serverless model hosting with caching, which is going to be the cheapest overall. Most of them also offer private AI deployment which is more expensive but your entirely own instance. Some of them more heavily quantize their models (deepinfra) so take that into your workload considerations. I included deepinfra because they tend to have some the new models sooner than others or have models the others don’t (like mimo). IMO, together > fireworks > deepinfra.
However, you’ll have to keep in mind that for true ZDR you give up caching which increases your inference costs, so it depends on how “Zero” you need the zero to be and on what timeframe. If you just don’t want your data being trained on, normal use is fine imo. If you are working on something actually sensitive (like, because of NDA agreements and such), then they might not be the right choice and you may be looking at more expensive offerings or building a machine.
One note: you have to pay attention to the source of the inference. Many (similar to open router) offer models that are pass-throughs to a providers API, but they are pretty clearly marked. Just make sure the model is marked as serverless when evaluating new ones. This is where a proxy comes in handy so you can prevent your LLM from accessing a different models from the specific ones you’ve vetted.
It used to be most of our spam came from Google accounts, and some still does, but now that is far and away outpaced by spam coming from Microsoft who does not take those same steps to deanonymize users. Also, “bot traffic” (AI traffic) is definitely a very real problem sites deal with. Also, you know who else uses Gemini from VPNs?
All the things you complain about seem to have real, direct consequences when not done.
I’m with you, I don’t want to be tracked and don’t want log in to every website. Just saying the reasons are pretty compelling when the reality we live in these days is a connected world with advanced technology where your average teenage could potentially hack a power plant and your average scammer can now use far more advanced AI tools to trick people.
This will only get worse, which is the real reason for the quiet US/EU push to unmask users, require IDs, etc. and not because some kids go online to naughty websites with boobs on them.
…and are not only “okay” with that, but enjoy it so much as to go online to ridicule others who don’t do the same? Weird.
News outlets always seem to forget “again” in the title.
It seems like your counter-point is “they already use everyone’s data, why didn’t you just assume they would”, which is a bit weird from a victim-blaming perspective but valid nonetheless.