Sounds like the lesson learned is using the right tool for the job -- reusing Kubernetes in an existing cluster to spin up sandboxes is a fair initial path to start offering the service. But Kubernetes likely isn't meant for rapid churn of workloads, here sandboxes.
The architecture to me seemed very similar to SeaweedFS [1] (Facebook Haystack [2]) except with an extra layer for sandbox-hosting nodes. Like requests go into a master, or the global load balancer, then to a volume server, which in turn knows where the files/sandboxes should go. There is no need for sandboxes to be managed with the Kubernetes overhead since the the nodes/bare metal servers probably have scheduling taints on them to preserve the memory/cpu for the sandboxes.
The architecture to me seemed very similar to SeaweedFS [1] (Facebook Haystack [2]) except with an extra layer for sandbox-hosting nodes. Like requests go into a master, or the global load balancer, then to a volume server, which in turn knows where the files/sandboxes should go. There is no need for sandboxes to be managed with the Kubernetes overhead since the the nodes/bare metal servers probably have scheduling taints on them to preserve the memory/cpu for the sandboxes.
[1] https://github.com/seaweedfs/seaweedfs
[2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...