I've done the same thing with Pi 3Bs/4s basically for the same reasons. I definitely ran into the limitations of the SDCards and set up USB drives and NAS storage class for better performance. I ended up running my own docker registry on my NAS and running all of the images through that before deploying to the k3s cluster. I also hooked up container scanning and automated it the deployments through ansible.
Things I would do differently are using NixOS or bootable containers (CentOS) (side note, bootable NixOS container would be a killer app) and writing my own helm charts instead of fully customizing my manifests and doing the deployments from ansible, and would recommend against Raspberry Pis for the compute as the 3's and 4's don't support limits, e.g. cpu or ram limits, and I wasn't able to set up firecracker containers correctly on the Pis.
I'm also exploring hyperconvergence infrastructure (HCI) as that seems more like my ultimate goal for homelab stuff.
Things I would do differently are using NixOS or bootable containers (CentOS) (side note, bootable NixOS container would be a killer app) and writing my own helm charts instead of fully customizing my manifests and doing the deployments from ansible, and would recommend against Raspberry Pis for the compute as the 3's and 4's don't support limits, e.g. cpu or ram limits, and I wasn't able to set up firecracker containers correctly on the Pis.
I'm also exploring hyperconvergence infrastructure (HCI) as that seems more like my ultimate goal for homelab stuff.