matrixgard

It is different from BI and the gap is worth naming. BI connects deterministic queries from a known operator schema. An agent is an unbounded query generator, so your risk surface includes both what it asks for and what it synthesises from what it gets back. RLS alone won't help you, because the moment the agent gets handed a view it is already reasoning over rows you wanted to protect.

The practical answer I've seen hold up: push column-level redaction before the agent layer, not after. A logical replica with PII columns replaced by null or a stable hash gives you the same query surface, plus one audit row per session at the connection pooler, not the app. The AI team gets its data, you get a hard boundary that doesn't rely on prompt engineering.

The harder question is ownership. In a startup where the ML lead, the infra person and the security person are often the same tired CTO at 10pm, the right answer depends on who gets paged when a hallucinated query wakes up the primary. Usually the answer is nobody, which is the real problem behind the technical one.

$34k in 8 days with zero users is the flavor of bug that makes CFOs distrust engineering. The thing that would have caught this: anomaly detection scoped to the service + the account, not just the total bill. Most teams monitor the aggregate and only spike-alert above some threshold — by then it's 4 days old. Per-service p95-vs-median alerts at hourly granularity would have flagged this inside 6 hours. Cloudflare should absolutely ship platform-side guardrails here, but until they do, self-built alerts at the service level are the only real defense.

eight months is usually when the first version of "what you built" stops working and you have to build the thing customers actually want. the gap between those two is what most people call "0 revenue growth."

we went through something similar building a fintech tool — spent months on features nobody asked for because we assumed we understood the problem. what actually changed it: stopped building for what we assumed and started building for one specific thing one customer said they'd pay for that day. just one. then the next.

what was the specific use case you landed on?

eight months is usually when the first version of "what you built" stops working and you have to build the thing customers actually want. the gap between those two is what most people call "0 revenue growth."

we went through something similar building a fintech tool — spent months on features nobody asked for because we assumed we understood the problem. what actually changed it: stopped building for what we assumed and started building for one specific thing one customer said they'd pay for that day. just one. then the next.

what was the specific use case you landed on?

Static IP whitelisting is a nightmare in practice -- we ran it for about 8 months and the support burden was basically someone's part-time job. Every time someone's hotel or coffee shop rotated IPs, they'd open a ticket. We moved to AWS SSM Session Manager a couple years back and haven't touched a bastion since. No open ports, no SSH keys to rotate, and `aws ssm start-session --target i-xxxxxxx` just works from anywhere as long as they have valid IAM creds. CloudTrail picks up the session automatically, which was a side benefit we hadn't even planned for.

IAM Identity Center (was just called SSO before the rename) with a short session duration -- we do 8-hour max with MFA at login -- handles the traveling employee case cleanly. They re-authenticate, no ticket. Overhead compared to running a VPN server you have to patch is basically zero.

The one thing that bit us: we kept a bastion sitting around "just in case" for way too long before we cleaned it up. It was live for almost a year after we didn't need it anymore. What's the main access pattern you're trying to solve -- DB access, SSH to EC2, or something different?

Deliberately is doing a lot of work in that sentence and it's exactly right. The shortcuts that kill you aren't the ones you knew were shortcuts it's the ones that felt like reasonable decisions under pressure and only look like shortcuts in hindsight.

The "isolate the risk" part is where most early CTOs underestimate the blast radius. A shortcut in the auth layer feels isolated until six months later when three other systems are built on top of assumptions it makes. By then it's not a shortcut anymore, it's load-bearing technical debt.

What's the recovery actually looked like in practice when you've caught it early enough — do you get clean rewrites or is it mostly containment?