luis_valerio

Is anyone using onprem open weigh models for programming? If so what models, gpus and feedback?

Thanks for sharing, do we know if is there any organization or production system really using it?

Best way to officially support shadow AI at government, sounds great idea

Yes but does it's not clear for me. It could be just a private interconnect to openai infrastructure. My guess is that is not hosted in gcp... That's why I'm asking

Understood yeap. Yes exactly is terraform/opentofu. The point is that we cannot do it at proxy level since we do allow any kind of operation attempt (we just filter that it's an infra req), then we generate the code based on the policy and, as you mention, once we have a validated plan we crosscheck against the compliance checker and provide an assessment about criticity and policy compliance...and trigger an automatic hitl for admin if needed otherwise it allows approve the pr an apply afterwards

To be precise the thing provide permission to do any operation, including destructive actions to the agent but we do set policies, auth controls and hitl force approval to validate the destructive or high impact actions before apply them

Really good point about using the same context for the compliance, currently it's a separate subagent but read rules/policies and then check against the other agent proposal, no possible to inject since we control totally via a rail the prompt for the compliance subagent but it's true that it checks policies so it's not deterministic (works pretty well but not deterministic) so that's we are targeting checking that policies deterministically with OPA and then the compliance will only add to the final result some other checks that need to be LLM inferences.

So it's pretty similar to yours, you have any other additional harness to be as much determistic as possible?

Hi, thanks for the detailed answer! Yes it's pretty similar to the approach we are following the only difference is that we cannot provide read permissions into the endpoint so we have some kind of compliance or auditor subagent that checks against forbidden actions and polices and as you mention also classify all the request in level of impact enforcing hitl for admins.

We are planning to do the policy check as much determistic as possible with OPA rules and so on, but we also have the policies in place at cloud level

Is it really gcp hosting openai in their cloud? Afaik is a third party marketplace so it looks like you're buying openai saas via gcp but not hosted in gcp. That's key difference with azure specially for egress costs if your data leaves gcp to openai right?

One question from the infrastructure side (i'm running a similar scenario but for infrastucture) where any mistake could not be reversible such as a deletion of a resource.

How do you handle an agent that has valid authorization but has been prompt-injected?