Comment by btaitelb

I don't see any ethical issue here, but if there are hard feelings involved (like you're leaving his investment high and dry), he could make things difficult for you at company B. I suggest talking to your new boss at company B, asking them the same question, and if appropriate for an introduction to the investor. Mr. X will likely want to know why you're leaving company A (is there something wrong with his investment?) and will appreciate that you're checking with him about ethical concerns.