That's an entirely reasonable skeptical point of view to take in the absence of other evidence. However, this question has also been examined. Using typical measures of socioeconomic status (e.g. an index of education + income), one finds that controlling for socioeconomic status does little to explain the black-white IQ gap. There is a sizable gap at every level of socioeconomic status. Data: http://sites.biology.duke.edu/rausher/Hm2.jpg
source: http://sites.biology.duke.edu/rausher/lec23_05.html