> Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.
I operated services at similar scale, and generally we use to put a bit of slack so that you would get an alarm when capacity goes up to 80%+ (or whatever number makes sense)
This allows to check, in the morning, after coffee, why the load balancer fleet didn't scale up automatically.
I am sure there is a good answer to why this is impractical, but it would be nice to know
I operated services at similar scale, and generally we use to put a bit of slack so that you would get an alarm when capacity goes up to 80%+ (or whatever number makes sense)
This allows to check, in the morning, after coffee, why the load balancer fleet didn't scale up automatically.
I am sure there is a good answer to why this is impractical, but it would be nice to know