Comment by ryao

malloc() should fail even with always over commit enabled if you try to malloc more memory than there is virtual address space, although that needs testing to confirm it. The alternative to failing would be hanging inside the kernel, which would be a DoS attack.