The problem is radiation. Empty space is not really empty, there are stray atoms floating around. Very scattered, but at a high impact momentum penetrate the ship ionizing anything in their way, and making the ship itself radioactive.
Not even talking about stray high-energy particles from distant supernovas and magnetars -- those irradiate ship regardless of its speed.
Not even talking about stray high-energy particles from distant supernovas and magnetars -- those irradiate ship regardless of its speed.