karmarepellent

The sarcasm is duly noted. But I simply answered the question. I don't have any strong opinion regarding passkeys.

A service that lets you sign up by uploading a SSH public key could just as well let you upload multiple public keys in your profile to be able to connect from other devices.

This is incorrect. SSH certificates work just like x509 certificates in that regard. Also, with PubkeyAuthentication, there exist all kinds of ways to collect host keys before connecting to them for the first time and thus avoiding the trust-on-first-use problem. Especially in private networks where you control all the nodes.

> Signing in is cryptographically signing a commitment to the current ephemeral tunnel.

I can see how SSH could be used for authentication on the web. And I have no doubt that it would be sound out-of-the-box. But I am not sure what you mean by your last sentence. Do you mean that authentication targets are gated and only reachable by establishing a tunnel via some kind of forwarding?

Aside from the wonderful possibilities that are offered by using port forwarding of some kind, you could also simply use OpenSSH's ForceCommand to let users authenticate via SSH and then return a short-lived token that can then be used to log into an application (or even a SSO service).

I guess no one uses SSH for authentication in this way because it is non-standard and kind of shuts out non-technical people.

I'm curious to know if people see this as a viable alternative to a PXE installation, especially when it comes to the deployment of large-ish (possibly air-gapped) clusters.

Actually I think the trick is to change ones own perspective on these things. Regardless of how many redundancies and how many 9's of availability your system theoretically achieves, there is always stuff that can go wrong for a variety of reasons. If things go wrong, I am faster at fixing a not-so-complex system than the more complex system that should, in theory, be more robust.

Also I have yet to experience that an outage of any kind had any negative consequences for me personally. As long as you stand by the decisions you made in the past and show a path forward, people (even the higher-ups) are going to respect that.

Anticipating every possible issue that might or might not occur during the lifetime of an application just leads to over-engineering.

I think rationalizing it a little bit may also help with the paranoia.

Its a matter of evaluating what kind of infrastructure your application needs to run on. There are certainly mission critical systems where even a sliver of downtime causes real damage, like lost revenue. If you come to the conclusion that this application and everything it involves better run on k8s for availability reasons, you should probably focus on that and code your application in a k8s-friendly manner.

But there are tons of applications that run on over-engineered cloud environments that may or may not involve k8s and probably cost more to operate than they must. I use some tools every day where a daily 15 min downtime would not affect my or my work in the slightest. I am not saying this would be desirable per se. Its just that a lot of people (myself included) are happy to spend an hour of their work day talking to colleagues and drinking coffee, but a 15 min downtime of some tool is seen as an absolute catastrophe.

I think it depends on the definition of "bricking the cluster". When you start to upgrade your control plane, your control plane pods restart one after one, and not only those on the specific control plane node. So at this point your control plane might not respond anymore if you happen to run into a bug or some other issue. You might call it "bricking the cluster", since it is not possible to interact with the control plane for some time. Personally I would not call it "bricked", since your production workloads on worker nodes continue to function.

Edit: And even when you "brick" it and cannot roll back, there is still a way to bring your control plane back by using an etcd backup, right?

Do you mind sharing what these operations were? I can think of a few things that may very well brick your control plane. But at the very least existing workloads continue to function in this case as far as I know. Same with e.g. misconfigured network policies. Those might cause downtimes, but at least you can roll them back easily. This was some time ago though. There may be more footguns now. Curious to know how you bricked your cluster, if you don't mind.

I agree that k8s offers many features that most users probably don't need and may not even know of. I found that I liked k8s best when we used only a few, stable features (only daemonsets and deployments for workloads, no statefulsets) and simple helm charts. Although we could have probably ditched helm altogether.

In theory: absolutely. This is just anecdata and you are welcome to challenge me on it, but I have never had a problem upgrading Kubernetes itself. As long as you trail one version behind the latest to ensure critical bugs are fixed before you risk to run into them yourself, I think you are good.

Edit: To expand on it a little bit. I think there is always a real, theoretical risk that must be taken into account when you design your infrastructure. But when experience tells you that accounting for this potential risk may not be worth it in practice, you might get away with discarding it and keeping your infra lean. (Yes, I am starting to sweat just writing this).

Agreed. The best thing we did back when we ran k8s clusters, was moving a few stateful services to dedicated VMs and keep the clusters for stateless services (the bulk) only. Running k8s for stateless services was an absolute bliss.

At that time stateful services were somewhat harder to operate on k8s because statefulness (and all that it encapsulates) was kinda full of bugs. That may certainly have changed over the last few years. Maybe we just did it wrong. In any case if you focused on the core parts of k8s that were mature back then, k8s was (and is) a good thing.

I think the value proposition holds when you are just getting started with your company and you happen to employ people that know their way around the hyperscaler cloud ecosystems.

But I agree that moving your own infra or outsourcing operations when you have managed to do it on your own for a while is most likely misguided. Speaking from experience it introduces costs that cannot possibly be calculdated before the fact and thus always end up more complicated and costlier than the suits imagined.

In the past, when similar decicions were made, I always thought to myself: You could have just hired one more person bringing their own, fresh perspective on what we are doing in order to improve our ops game.