Comment by sarnowski

You absolutely can and if you know that you don’t install software that needs it, there is no reason to install X during installation.