Comment by dev_hugepages

This is called an oomkiller. The kernel has one but it kicks in very late and the kernel prefers to do page trashing instead of killing processes.

systemd-oomd should be integrated in systemd, you can configure it to your liking and see if it improves your problem.