Comment by alpb

During many years of operating several-thousands of nodes production clusters on Kubernetes, I've never seen any of these observability tools that query kube-apiserver work at that scale. Even the popular tools like k9s make super expensive queries like listing all pods in the cluster that if you don't have enough load protections, can tip your Kubernetes apiserver over and cause an incident. If you're serious about these querying capabilities, I highly recommend building your own data sources (e.g. watch objects with a controller and dump the data in a sql db) and stop hitting apiserver for these things. You'll be better off in the long run.