We can duplicate the "Global Query Layer" in multiple geo regions with their own replicated Grafana instances. If a single region/zone has trouble we can still access metrics in other regions/zones. This avoids Thanos having any SPoFs for large multi-user(Dev/SRE) orgs.
With our current Thanos deployment, we can tie a single geo regional deployment together with a tiered query engine.
Basically like this:
"Global Query Layer" -> "Zone Cluster Query Layer" -> "Prom Sidecar / Thanos Store"
We can duplicate the "Global Query Layer" in multiple geo regions with their own replicated Grafana instances. If a single region/zone has trouble we can still access metrics in other regions/zones. This avoids Thanos having any SPoFs for large multi-user(Dev/SRE) orgs.