Barring an extremely small number of cases, this is not true by any means. The cognitive skills required to learn math are the same as the cognitive skills required to learn language. If you're interested in learning more about this, check out Keith Devlin's "The Math Gene" [1]
[1]: http://www.amazon.com/The-Math-Gene-Mathematical-Thinking/dp...