I would be interested to know what storage driver they're using for their nodes. High container churn puts a lot of stress on the VFS subsystem of Linux, and we've seen cases where customers have trigger lots of mount/umounts which results in filesystems causing panics. At SUSE, we do have some kernel devs debugging the issues, but the workaround is almost always "rate limit all the things". There are a few other kernel areas that are stressed with high container churn (like networking), but VFS is the most likely candidate from my experience.
While on paper containers are very lightweight, spawning a lot of them exercises kernel codepaths that probably haven't been exerted to that type of stress during development.
While on paper containers are very lightweight, spawning a lot of them exercises kernel codepaths that probably haven't been exerted to that type of stress during development.