hsin003

have you tried lxc + eBPF? the former allows you to have lighter & faster (share kernel) and the latter gives you security. that is what we do for containarium.

We hit a Cloud NAT bill of ~$4,500/month (3.2 TiB/day at $0.045/GiB) on a project where we'd "enabled Private Google Access" on the subnet. The traffic was inference workloads pulling data from GCS — exactly what PGA is supposed to put on the private path for free.

The host is a multi-tenant LXC setup (Containarium — open source, many isolated user containers on one VM behind a shared NAT IP). The cost-saving reason we run it that way is exactly the same reason this incident was sneaky: ~20 workloads sharing one egress IP means VPC flow logs and Cloud NAT metrics all point at the same place, with no native GCP way to attribute traffic per workload.

Where the platform earned its keep: it tracks per-container resource stats (CPU/memory/disk/network via veth counters) and exposes them in a web UI. We sorted containers by lifetime tx bytes, and the offender jumped out — xxx-container, 1.5 TB rx since boot. Two orders of magnitude above any other container.

Without per-container traffic accounting, we'd have been left correlating VPC flow log timestamps against ps and lsof on the host — the kind of investigation that takes a day, not five minutes.

Then we looked at the destination IP from VPC flow logs: 192.178.163.207 → tt-in-f207.1e100.net. Google-owned. That misled us at first: it looked like PGA was working, just to a different Google service. It wasn't.

The actual problem: PGA has two halves and GCP only surfaces one of them in the subnet UI.

1. Subnet flag — --enable-private-ip-google-access. We had this on. 2. DNS — *.googleapis.com has to resolve to the private VIP range 199.36.153.8/30 (or 199.36.153.4/30 for restricted). Without that, storage.googleapis.com resolves to a normal public Google IP, the route to PGA never gets used, and Cloud NAT processes every byte.

The fix is a Cloud DNS private zone for googleapis.com attached to the VPC, with A and AAAA records pointing at the private VIP range (don't forget IPv6 — we hit that on attempt one and saw traffic go right back out the public path). Once that's in place, dig storage.googleapis.com from inside the VPC returns 199.36.153.11, traffic uses the private path, NAT bytes drop ~95%.

Verification one-liner from a VM in the VPC:

$ dig +short storage.googleapis.com 199.36.153.11 # private VIP — good # vs. 142.251.x.x # public IP — your "PGA-enabled" subnet is doing nothing

The annoying part is that everything else looks correct. Subnet flag set, no external IP on the VM, destination is a Google IP, NAT gateway is healthy. There's no warning anywhere that DNS is the missing piece.

Worth checking on every VPC where you assumed PGA was doing its job.

I agree CI should catch as much as possible — image scanning and dependency checks at build time are table stakes.

But in practice, CI is only a point-in-time guarantee. A build can pass all checks and still become vulnerable later as new CVEs are disclosed.

So the goal isn’t to rely on runtime to “catch mistakes”, but to add a second layer of defense — continuous monitoring and probing for already-deployed services.

If anything, this incident showed us that CI alone isn’t sufficient once systems are long-lived.

I think both points are true in practice. Reviewing AI-generated code can require more experience than generating it, but at the same time some basic checks (dependency versions, release notes, etc.) are still worth doing.

One thing this incident reminded us of is that review is only a snapshot in time. Even if everything looks fine when a PR is merged, new CVEs can appear later and suddenly make previously safe dependencies vulnerable.

That’s why we started treating monitoring and vulnerability checks as part of the platform itself, not just the review process.

That’s a great point about fresh-context reviews — the same session that generated or assembled the code often won’t catch its own mistakes.

What worried us in this incident is that even if you catch everything during review, new CVEs can appear later and suddenly make previously “safe” code vulnerable.

That’s why we started treating monitoring as part of the platform itself — every service runs with centralized checks so unusual behavior (CPU spikes, unexpected processes, endpoint exposure) gets flagged quickly.