Aside from the usual Magic Dirt Theory, there is also the Magic (American) Racism Theory. James Flynn explains the conundrum for environmentalists:
There is no doubt that high h² [heritability] estimates force environmentalists to find a factor or factors that are relatively uniform in their presence within the black population – and within the white population as well if they operate there. After all, if an environmental factor is potent enough to account for the 15-point performance gap between black and white, and if it varies much from person to person within the black population, it would be extremely odd if it accounted for none of the variable performance within the black population! And if it did, it would of course increase the role of environmental factors in explaining IQ variance and thus lower the h² estimate for blacks. There is also no doubt this criterion, the criterion of uniform presence, is the most crippling of those the environmentalist is forced to accept. If we seize on SES as a between-population explanation, who can deny that there are large differences in SES within black America; if we seize on education, who can deny that blacks differ significantly in terms of quality of education? The usual candidate brought forward for the role of blindfold is racism: after all every black suffers from racial bias, and no white suffers from at least that kind of handicap, and racism is very potent. But this too is simply an escape from hard thinking and hard research. Racism is not some magic force that operates without a chain of causality. Racism harms people because of its effects and when we list those effects, lack of confidence, low self-image, emasculation of the male, the welfare mother home, poverty, it seems absurd to claim that any one of them does not vary significantly within both black and white America. Certainly there are some blacks who have self-confidence, enjoy a stable home, a reasonable income, good housing; and certainly we all know whites who have a poor self-image, suffer from emasculation, or suffer from poverty. [p. 59-60 in Race, IQ, and Jensen, 1980]
This illustrates what Jensen wrote back in 1973:
The very ad hoc nature of environmentalist explanations seems to me antithetical to the ways of science. Scientific progress is won through an unrelenting battle against ad hoc explanations of natural phenomena. Therefore, in studying subpopulation differences in mental abilities, does it not seem a more scientific approach to consider all factors which are known to cause individual differences within groups? And is it not reasonable, if for practical reasons of research strategy we must assign some priority to the hypothesized causes we wish to consider, that the evidence derived from studies within groups should serve as a guide to the kinds of hypotheses most worth entertaining about the causes of differences between groups? And does not this lead us directly to the hypothesis of genetic factors as being among the undoubtedly multiple causes of racial subpopulation differences in mental abilities? Further more, it is practically axiomatic in biology that any characteristics showing individual variation within subgroups of a species will also show variation between subgroups of the species. [p. 129 in Educability and group differences, 1973]
Jensen (p. 509ff in The g factor, 1998) nevertheless provides a reply to people who offer Unique American Legacy of Slavery model of African American-White IQ gap:
Those behavioral scientists who attribute the difference entirely to the environment typically hypothesize factors that are unique to the historical experience of blacks in the United States, such as a past history of slavery, minority status, caste status, white racism, social prejudice and discrimination, a lowered level of aspiration resulting from restricted opportunity, peer pressure against ‘ ‘acting white,” and the like. The obvious difficulty with these variables is that we lack independent evidence that they have any effect on g or other mental ability factors, although in some cases one can easily imagine how they might adversely affect motivation for certain kinds of achievement. But as yet no mechanism has been identified that causally links them to g or other psychometric factors. There are several other problems with attributing causality to this class of variables:
1. Some of the variables (e.g., a past history of slavery, minority or caste status) do not explain the W-B 1 sd to 1.5 sd mean difference on psychometric tests in places where blacks have never been slaves in a nonblack society, or where they have never been a minority population, or where there has not been a color line.
2. These theories are made questionable by the empirical findings for other racial or ethnic groups that historically have experienced as much discrimination as have blacks, in America and other parts of the world, but do not show any deficit in mean IQ. Asians (Chinese, Japanese, East Indian) and Jews, for example, are minorities (some are physically identifiable) in the United States and in other countries, and have often experienced discrimination and even persecution, yet they perform as well or better on g-loaded tests and in g-loaded occupations than the majority population of any of the countries in which they reside. Social discrimination per se obviously does not cause lower levels of g. One might even conclude the opposite, considering the minority subpopulations in the United States and elsewhere that show high g and high g-related achievements, relative to the majority population.
3. The causal variable posited by these theories is unable to explain the detailed empirical findings, such as the large variability in the size of the W-B difference on various kinds of psychometric tests. As noted in Chapter 11, most of this variability is quite well explained by the modified Spearman hypothesis. It states that the size of the W-B difference on various psychometric tests is mainly related to the tests’ g loadings, and the difference is increased if the test is also loaded on a spatial factor and it is decreased if the test is also loaded on a short-term memory factor. It is unlikely that broad social variables would produce, within the black and white populations, the ability to rank-order the various tests in a battery in terms of their loadings on g and the spatial and memory factors and then to distribute their effort on these tests to accord with the prediction of the modified Spearman hypothesis. (Even Ph.D. psychologists cannot do this.) Such a possibility is simply out of the question for three-year- olds, whose performance on a battery of diverse tests has been found to accord with Spearman’s hypothesis (see Chapter 11, p. 385). It is hard to even imagine a social variable that could cause systematic variation in the size of the W-B difference across different tests that is unrelated to the specific informational or cultural content of the tests, but is consistently related to the tests’ g loadings (which can only be determined by performing a factor analysis).
4. Test scores have the same validity for predicting educational and occupational performance for all American-born, English-speaking subpopulations whatever their race or ethnicity. Blacks, on average, do not perform at a higher level educationally or on the job, relative to other groups, than is predicted by g-Ioaded tests. An additional ad hoc hypothesis is required, namely, that the social variables that depress blacks’ test scores must also depress blacks’ performance on a host of nonpsychometric variables to a degree predicted by the regression of the nonpsychometric variables on the psychometric variables within the white population. This seems highly improbable. In general, the social variables hypothesized to explain the lower average IQ of blacks would have to simulate consistently all of the effects predicted by the default hypothesis and Spearman’s hypothesis. To date, the environmental theories of the W-B IQ difference put forward have been unable to do this. Moreover, it is difficult or impossible to perform an empirical test of their validity.
This explanation has not seen any serious work since these words were written. In fact, it has sort of failed even more with the decline of stereotype threat theory. Granted, there is not yet a large pre-registered replication for the black-white version (YET, there is a pre-registration!), but there is an excellent one for the male-female math version (Flore, 2018). And there is a meta-analysis of stereotype threat for black-white with publication bias present, and some large sample studies already (not cited much of course).