My experience with stuff with C-extensions (although not specifically ML packages) is that there is almost always a wheel published for them these days, so pip works just fine and I almost never have to think about if something is pure python or not.
I wish I could say I never had to think about it, though.
I wish I could say I never had to think about it, though.