If you actually want to know 'why' people think so, have a look at all the discussions around insider trading in financial markets. No one is forcing you to buy stocks at gunpoint, either.
(And, yes, there are good arguments in favour of allowing insider trading in all markets, and a few against.)
(And, yes, there are good arguments in favour of allowing insider trading in all markets, and a few against.)