I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time, so for large closures you give up a lot of the benefit of the cluster being able to cache layers (as in, say, a nix2container- or nixery-style image) and achieve very fast subsequent startups.
I'm a k8s novice, but would there be a way to run the nix daemon/cache in a semi-persistent pod on each node, and then "attach" it to the actual worker pods?
I'm a k8s novice, but would there be a way to run the nix daemon/cache in a semi-persistent pod on each node, and then "attach" it to the actual worker pods?