The working model for this kind of work is that a simplified model of human inequality approximately holds. In this model, every group of people has a distribution of and thus also an average value for human capital traits, chiefly but not exclusively intelligence. These distributions are primarily caused by genetic differences, as in, (true) polygenic scores (genetic potential) for the relevant traits. When people move around, they keep this genetic value, and this is true also for groups. In recent history (<100 years), the values are approximately constant though not in the more distant past (100 or 1000s of years). Thus, the basic prediction here is that if we can estimate this average human capital of particular groups, this should predict their performance when they move (migrate) or mix with others. Here one must take into account selection effects, that is, that migrants are not necessarily and not generally representative of the home habitat/country in these human capital traits. Thus, it is necessary to measure such selection effects. Following the initial work by the IAB on their brain drain dataset, I’ve used completed formal education level as a proxy for this. These estimates were generally obtained from ‘microdata’ (individual data) that were matched with the same cohort and age group at home (insofar as possible), counting only migrants who arrived as adults (25+), so that degrees earned in the host country are not counted. Thus, if a population at home has e.g. 10% of people with university degrees, but 70% of migrants have this, then they are very selected. The most selected corridors are:

Extreme selection is mostly seen for very poor origin countries, which is only logical, since the lower the education level in the origin, the larger the difference can be. Many poor countries have university degree rates in single digits, but manage to send people to the west with 50%+ such degrees. They are educating only a few people in university, and large numbers of them are leaving. People who are high human capital but leave their countries before taking education would thus generally be misestimated.

This kind of way of measuring selection only works for adults with completed education, a major limitation, so take it as a rough proxy and nothing more. It is possible to genetically measure the selection effects by comparing home/representative samples to migrant samples, though this has never been done systematically to my knowledge. This is generally because samples are too small or the current polygenic score estimates too imperfect (especially for non-Europeans). Performance of a given group in some host country is measured using whichever available social and cognitive data including employment rate, income, educational attainment, cognitive scores (e.g. PISA), crime, and social welfare use. Prior research has shown that it is not generally important which specific metric is used since the groups rank about the same across them. This working model is not entirely true, especially because some social metrics are strongly affected by age/sex composition (e.g. crime), time in the country, language/cultural distance (e.g. test bias). Thus, it is preferable to study 2nd and later generations. The problem with those data is in general that many countries do not provide useful data. This is especially true for people who change their citizenships, and children from mixed relationships (e.g. German mother, Spanish father). I have been inclusive and also used data based on citizenship when country of birth or parental country of birth was not available. This causes some biases since naturalization rates differ by group systematically: people from worse countries are more eager to convert to a new superior citizenship, especially because it generally means they cannot be deported (so easily). Countries also differ in how easy it is to obtain these citizenships (e.g. automatic at birth for many New World countries), which further adds noise and bias to the results. With these issues in mind, we don’t expect perfect results and neither do we find them.

So the basis of the analysis as of writing are data from these 34 host countries:

They are color coded by how much and how useful data they provide. In general, larger countries provide estimates for more origin groups owing to larger migrant populations. Countries with better government databases provide more data, insofar as the results aren’t completely censored/hidden. I have mostly obtained results from Western countries because these have high rates of immigration in the last few decades as well as useful public data. It’s possible there are more useful data to be found in e.g. Thailand. Belgium was the only western country I was unable to find any useful results for, perhaps a reader can find something.

The overall results for the origin countries should under the working model reflect the average human capital in the people from those countries, and the map we get looks like this:

Thus, it looks about the same as every other such map. Even after adjustment for observed migrant selection, Mexicans, South Africans, Indians do fairly well. You can click any country to see what data the results are based on. E.g., for Mexico:

So we can see that Mexicans don’t do well in USA or Canada, but do well in Europe. If you check migrant flows, something like 95% of all Mexican migrants live in the USA and are relatively unselected. A small number are in Europe, much more selected with resulting good outcomes. Depending on how you weight the results, Mexicans thus may obtain a large negative (if using migrant size weights), or somewhat to strongly positive score (if no weights). The overall models change somewhat depending on the weights, but not in a consistently obviously better direction, so I’ve used a middling approach to weights.

If we take the current ‘best estimate’ for each origin group and correlate it with national IQs (the best human capital measurement) or home country general development (HDI but better), we get:

And:

Thus, the main prediction of the working model is seen in the data. Results are not perfect, of course, given all the issues with the data quality, measurements, time-in-country, cultural/language biases in tests, and imperfectly modelled selection effects. But they are pretty good and compelling. Now no one can credibly claim that origin country metrics do not predict migrant outcomes consistently.

Most tests show male advantages, and only 3 show a female advantage, all of them quite small, below 0.2 d (multilingual vocabulary, color sorting, eye reading). By contrast, many of the male advantages are quite large, many above 0.5 d. These results thus suggest a general male advantage but focused on certain kinds of tests. The differences in these less general abilities than general intelligence are not surprising or new, though occasionally denied or questioned by feminists (e.g. male advantage in math). However, there is another potential explanation. The users on the site are obviously not representative but rather self-selection for high intelligence. It is possible that the male takers are more selected than the female ones.

To get around this somewhat I decided to analyze the large public datasets that the OECD has published from their PIAAC studies. This is the adult version of PISA. There’s data on 3 tests across 2 waves, and the gaps look like this, pooled across countries:

The difference in literacy was ~0, men scored about 0.20-0.26 better on math, and 0.08-0.10 d better on ‘problem solving’. Thus, again, consistent with a general male advantage. Obviously, the OECD aren’t big fans of Richard Lynn, so you can be quite sure they tried to get rid of these gaps in their item designs, but didn’t quite make it. For good measure, I ran my own item response theory models and checked for test bias, though it was quite minor.

The gaps also vary quite a bit across countries, wave 1:

Wave 2:

Insofar as age is concerned, the two waves don’t agree entirely:

In the first wave, the male advantage appears to be growing with age, but not in wave 2. I am not sure what would be the cause of this. Usually these kinds of abrupt changes are due to changes to the tests, mode of delivery, scoring, and not real differences. The datasets are far too large for these differences to be due to random sampling error. The cohort (birth year) gaps seen between the waves suggest a change in measurement or sampling.

From a Richard Lynn perspective, the findings aren’t that surprising in general, the male advantage overall would be quite small, an unweighted mean of about 1.5-2 IQ. His general proposed model concerns how to resolve the question of why we see a much greater male brain size, which is seen in every population, and which remains substantial controlling for height, with the known causal effect of brain size on intelligence, with supposedly no difference in general intelligence. This would only be the case if women have some compensating brain advantage, or men’s extra brain size occurs in areas unrelated to intelligence. As far as I know, there is no known candidate for such a female advantage, and brain imaging studies find that the entire brain, more or less, is involved in intelligence. So Lynn’s solution is that researchers had just made an error and men really do have such an advantage in line with the brain size advantage. This resolves the overall problem, but creates a new one. If men are smarter in general because of a larger brain, then why do we see such variation in the size of this advantage? It doesn’t make a lot of sense if this advantage has deep ancestral roots and should be more of a human universal than something that varies between populations and their cultures. I’m afraid I don’t know what the resolution to this question is.

Thus, about 28% of runs result in victory, which also varies a bit by character (highest is 30% for Defect, lowest is 26% for Regent). When playing, you are wondering what is the best decision to make in a given situation. By utilizing a large number of games, it is possible to give some causal answers to some of those questions. Each run also has quite a lot of randomness, so it is possible to score a given run for luck. Some card options are better than others, some relics worse than others. The top players manage to win nearly 100% of their runs, but average Joe not so much. The top players have built tier lists, so we can compare those to empirically obtained results.

Relics

First off, there’s primarily 2 things to look at: cards and relics. Both can be obtained from combat rewards, shops (by choice and availability), and events. The goal is to predict which build — character+deck+relic combination — wins. Starting with the simpler of the two, relics, the most obvious approach is to check which particular relics are commonly found in winning vs. losing runs:

But you can probably see that it’s a biased approach. The issue is survivorship bias. Runs that lose will end up with fewer relics since they miss out on opportunities to get more of them later in the game. It can also clearly be seen in the transforming relic Sword of Stone, which becomes Sword of Jade when 5 elites have been killed. Losing runs commonly die before doing so while winning runs do so, thus giving the opposite effects of the pairs. One can also see the bias by just comparing the relic counts:

So this approach will not clearly tell us what we want to know but we can get more sophisticated. What if instead of looking at finished games, we look at them at some earlier point, say, after the first boss (floor 17). This limits us to the runs that made it that far (64%) while keeping their relic variation and predicting the future from that point. This probably works reasonably well. Given the amount of data we have, we can also fit a single logistic regression with every relic in the game and by character:

We see some expected results. Skills are generally too strong in the game, and the relic that makes any acquired skill upgraded is a top relic for every character (Toxic Egg). And for a replication of sorts, we can do the same thing at another point, say, after boss 2 (floor 33):

The models additionally controlled for the difficulty (ascension) and player (fixed effect). Still, the models don’t guarantee a causal result because relics already picked up until that point affected the outcome. Some choice remains, and thus the possibility of confounding. Fortunately, we have some other options available. There are a number of different things that happen in the game that assigns a random relic to the player, which is thus uncorrelated with anything. Using this variation, we can see clearly. Random relics come from starting relics (Neow), elite rewards, chests, and some events. Each of these can be used to figure out the causal effect of each relic. The estimates they produce should agree with each other perfectly, random sampling error aside, and they seem to do so:

Here we see that the random relic models agree with each other relatively close to perfectly, while the cross-sectional data models even with controls are a little off. For some strange reason, Lava Rock random relics do not agree much with the rest. Based on these results, we can meta-analyze the signal from the different approaches to get a single best estimate for each relic x character combination:

Of particular interest are the boss/ancient relics, which are given after killing a boss. A particular ancient is shown and offers some relics in a limited pool. These can be relatively precisely estimated using the instrumental variables approach. Relics can only affect your winning chances if you are actually offered them, so we get the following causal path: offered -> picked -> win. The picking rates are affected by players’ choice based on their guesses about which relic gives the best winning chances, so it is not a clean causal estimate. However, the offers are random, so they allow us to get around this. This gives us the following results (pooled across characters):

It is likely that these differ by character but even 400k games is not actually enough to precisely estimate it, believe it or not. That’s because the outcome is a binary, and we need a standard error of, say, 2%points. This requires about 26,000 games with that character, but we have around 49,000 (for act 2) and 30,000 (for act 3), which are further split by the ancients appearing. More data is actually needed.

To be noted, shop-only relics are difficult to estimate. The only method I can think of is using the randomization from Lord’s Parasol (which gives every relic in shops). However, there weren’t enough runs to precisely estimate these.

Furthermore, some of the ancient relic options are not independent:

Thus, these present issues with estimation as there is never a choice, for instance, between X and X.

Cards

Moving on to cards, we can do the same thing as before. We can try the simplest model of comparing cards found in winning vs. losing runs. It gets more difficult here because decks can have multiple copies of a card, and they can be upgraded or not. I’ve decided to merge the data for the regular and upgraded cards to boost precision at some loss of detail. With enough data, one could separate the two, and figure out which cards are most important to upgrade. With regards to the counts, I’ve tried binary, count, and capped count approaches (deck has card yes/no, deck has X copies, deck has 0,1,2,3+ copies). The capped counts work best. The issue with the full count is that the rare boss/ancient relic Pael’s Growth makes it possible to clone cards, so a few decks end up with literally thousands of copies of some cards which mess up the results (meme builds). Anyway, these are the simplest results:

We can note a limitation of the data here, Mad Science is actually 9 different cards, but the save files don’t note which one is which. Most of the very bad cards are actually curses or quest cards (Lantern Key). We can also move on to post-beating boss 1 decks to avoid most of the survivorship bias:

It seems taking the Spoils Map is generally overly greedy, which is presumably why you often see the pros skipping it.

These estimates are still not entirely unconfounded, but we can utilize the final approach, which is to use instrumental variables for card rewards. Since cards can only affect your winning chances if you actually picked them, we get the following causal path: offered -> picked -> win. This allows the use of instrumental variables to estimate the causal effects despite players having a choice in what to pick. The correlation matrix for the card estimates across methods:

Aggregating the randomization-based estimations into a single best estimate, we can figure out that these are the best and worst cards for the characters:

Player choices

We have lots of data on what people actually pick in the game when given a choice. Usually, from a fight reward, 3 cards are offered but only 1 can be picked. So it is possible to rank cards by players’ preference. The simplest metric is just picked/offered ratio. More sophisticated is using 3+1 variant of the Elo rating. It turns out these produce about the same result, so it is not worth much except to highlight the value of skipping later in the game (the smaller your deck is, the more likely you will draw your best cards). Thus, we can compare players’ preference with our best estimates of what is best:

Relics are generally acquired randomly, so there is not much choice to look at. Except for shops and ancients where players choose themselves. So do players tend to buy the best relics?

It would appear overall players are fairly rational or tuned in to what is good. But critically, does it relate to player skill levels? We expect that the better players would buy the better cards and relics. To do this analysis, we have to estimate player skill. The model is: win ~ player_id + ordinal(ascension) + character. Using these, we can plot the buy rates vs. causal estimates as a function of player skill:

The findings confirm the expectation, higher skilled players’ choices correlate more with our best estimates 0.56 for high skill vs. 0.36 for low skill in case of relics. The difference is much smaller for cards, suggesting that the main separation between the good and the bad players is knowing what relics to buy in shops.

Conditional effects

The main issue with the above is that what is best for a build to some extent depends on what you already have. Some cards are good in all builds, while others are bad in some builds and good in others. The same for relics, though they are usually never bad for you (giving only positive effects, some exceptions like Royal Poison or Brimstone). Statistically speaking, such conditional effects are interactions, and we are not too keen on interactions on this blog. Still, there are surely lots of real interactions here, given that the very purpose of the game is to know about these and use them skillfully to win. There is, however, a severe statistical issue. With about 83 distinct cards per character, there are 3,403 different 2-way interactions to model for an exhaustive search (for a total of 17,015). Even limiting ourselves to within character combinations (it is possible to obtain cards from other characters as well as colorless/neutral cards), the combinatorial explosion results in severe statistical penalties for finding them. So we have to use a bit of theory or other empirical data to find them. Fortunately, we can use the winning builds data here. Each character has a number of primary build types, or archetypes. These can be statistically found by looking at the winning decks, in this case for Silent:

We see roughly the expected types: shivs, poison, draw/discard/sly. Most decks are not pure because it isn’t generally possible to win and never pick a suboptimal card in the final deck, and many cards seem to go in just about any winning build because they are just good overall (e.g. Adrenaline). A few also go into multiple (Reflex goes with Hidden Daggers). The results for Haze are based on the old version where it triggered on discard.

Based on these plausible pairs of cards, we can test all of these 2-way interactions specifically, and for each character. The model is: at floor X, after winning normal combat, conditional on already having X in your deck, does it predict winning chances to see Y? We call these positive interactions synergies:

All in all, pretty credible causal estimates could be made in most cases. The combinatorial explosion seen for cards prevents us from searching too hard, at least until more data is collected.

General findings

In general, the adjusted cross-sectional models agree pretty well with the randomization tests (IV or not), so in a way, using the more fancy methods was not necessary. This is of course only in hindsight we can say this. And the estimates don’t agree perfectly even after adjustment for random errors, so there is likely some confounding left in the cross-sectional estimates. We also confirm that our best estimates are more in line with what players do.

All the more detailed results for all cards and characters can be found on the site now.

The analytic strategy was about the same but expanded a bit:

1. We pick 3 random variables, set the first as the outcome, and the other 2 as the predictors, thus getting the general model Y ~ X1 + X2. This we test against the interaction model Y ~ X1 * X2. The variables are continuous variables, occasionally quasi-continuous, so we don’t have to use anything other than OLS
2. In variants of the above, we swap X2 for sex (binary), since sex interactions are probably those that are most common/largest.
3. We use LOOCV to assess how much variance the interactions add. This is probably better than trusting the analytic approximations (adjusted R²).

Alright, so did this dataset change the conclusions? Not really. We find just about the same as before:

Or if you prefer a table:

So in a sizable number, even a majority, of models the variance explained % gain was negative, which is to say that there was so little signal of an interaction that model overfitting caused the gain to be negative. Some somewhat sizable interactions were found in about 1% of the models. But if you look more closely at them, you can see the problem is not an interaction but that the main effects are misspecified as linear when they are really nonlinear:

In this case, the model misfit is also caused by a data dependency. Someone measured at age 20 cannot have their age of heaviest weight at more than 20. It’s a kind of duh finding and not what people have in mind when they claim interactions.

I thus redid the models adding splines for the main effects and still testing for linear interactions (Y ~ spline(X1) + spline(X2) + X1:X2). Allowing for nonlinear main effects further reduced the apparent effect sizes of the interactions:

But this still leaves the possibility that the interaction itself is nonlinear (Y ~ spline(X1) * spline(X2)). Here’s a clean case of that happening:

Men’s level of estradiol is basically constant throughout their life (flat line fit), while women’s follow the fertile window (requiring a spline), and thus the interaction also requires a spline. Thus, redoing the models once again with nonlinear interactions allowed:

Given the extreme model flexibility, it is very apt to overfit even on large datasets, and 87% of non-sex models were worse than without the interaction. After all of this, something like 0.6% of random trio models, and 2% of random trio models with sex added at least 1% variance explained. Most of these are still going to be marginal. Like this one:

Looking closely, you can see the lines diverge slightly at the high end. And this is a relatively large real interaction effect!

As such, the results from the prior study cannot be explained away as being due to the nature of the dataset used, unless of course you can come up with another ad hoc hypothesis for why it also fails in NHANES, and presumably why interactions would show up in other kinds of data. These 2 studies cannot rule out that you can find some kind of dataset where practically relevant interaction effects show up (say, gene expression levels or other low-level biological systems). For now, however, the burden of proof is on those thinking interactions are commonplace and practically meaningful. Meanwhile, whenever you read a new study claiming some interaction effect with p = 4% based on a sample of 100 students, you are right to be very skeptical indeed.

In real life, however, interactions are mostly not a real thing. This is despite their sometimes great intuitive appeal and well-known examples (e.g. mating success/attractiveness as function of height and sex and their interaction). So I decided to hammer home this message so that there is something robust and convincing one can refer to to prove the point. For this purpose, I reached for my trusty revolver OKCupid dataset. It’s very large, and has an extreme variety of items. Yes, it is from a dating site, but every other well-known finding I’ve tried replicating in the dataset worked fine. The very fact that it works so well, as do other convenience samples, is evidence against the importance of interactions. So here’s what I did:

1. Item-level data: pick a random ordinal variable as the outcome (Y), and 2 others as the predictors (X1, X2). Then fit the additive and interaction models: 1) Y ~ X1 + X2, vs. 2) Y ~ X1 * X2. Repeat 1000s of times. This approach needs ordinal regression since the outcomes are ordinal (all items have 2-4 answer options, which in the base case are ordinal or binary, sometimes nominal).
2. Scale-level data: aggregate items into scales from my recent 10-dimension model using a mixture of item types (binary, ordinal, nominal), then repeat the same approach as above using scales instead of items.
3. Scale-level data: aggregate items into scales that I had previously investigated in the cat-dog study, then repeat the same approach as above. The scales here are quite different from the ones above as they were handpicked to measure particular dimensions of interest instead of ‘falling out of’ the data.
4. Variants of the above swapping X2 for sex, to specifically look at sex interactions, the most intuitively plausible subset.

First we have to think of what we care about. The most obvious is to look at the p-values. This was the focus of the prior studies on interactions in this dataset. It is, however, a mistake I think. The dataset is so large that many terms reach p<5% even with tiny effect sizes. What we really care about is not so much whether there is literally any signal larger than pure noise, but whether these signals are of practical concern. For this reason, here we will focus on changes in model explanatory power, that is, R² values (which can be deceptive but bear with me).

Alright, so what was found? It can be summarized in a few ways graphically:

In general, random models do not explain a great amount of variance in the outcome, but they do explain more than 0. Everything, after all, is correlated if you look close enough. For the item models, the median additive model explained 0.67% and the interaction model explained 0.67% for a relative gain of 0%. For the 2 sets of scale-level analyses, the results were about the same. For the 10-dimensional scales, the median additive model explained 2.83%, and the interaction model explained 2.97%, or a relative increase of 4.8%. For the replication using the cat-dog set of scales (general intelligence, mental health, antisocial behavior, prudence, conservatism, extroversion, reading, enjoys discussion, religiousness, drugs, kink/sex), the median additive model explained 5.61% and the median interaction model 5.64%, for a gain of 0.008%. However, even these values can be considered overestimates. The medians don’t come from the same models. If one compared within the models (trios), then the relative % gains are -0.3% for the items, 0.1% for the cat-dog scales, and 1.2% for the 10 dimensions. So a typical model that explains 2% of variance without interactions grows to something like 2.01%, 2.002%, 2.024%. Practically useless. For the models with sex, they did slightly better, but not much better in absolute terms. The table below summarizes the numerical results:

Alright, but since we already looked through the data to find interactions, what were the largest ones that were found? For the item-level analysis, they look like this:

It is somewhat tricky to understand the plot since there are 3 moving parts. The lines are the 2 sexes in their appropriate colors. The interaction term is always sex, and 4 of the 4 involve changing your name upon marriage, traditional female behavior. Thus, conservative women will lean towards “yes” and conservative men towards “no”. Any question that relates to conservatism thus works to get this effect (male gallantry, flag burning, watching sports on TV, mate guarding). These examples are somewhat trivial, as when I found that the largest effect of your astrological sign was predicting which season you like the most (few people prefer winter, but those born in the winter prefer it slightly more than average). For the scales, the largest interactions were:

These are basically the same thing 4 times. It’s a measure of conservatism, a measure of religiousness, and a measure of libertinism (sex or drugs). These effects are also relatively minor, the largest added an additional 1.25% variance but the main model already explained 25%, so the relative gain was only 5%.

One limitation of the above is that the measures aren’t perfectly reliable. Mathematically, in practice interactions also suck because interaction terms have the reliability of the product of the terms, so if both main variables have a semi-respectable reliability of 0.80, the interaction term is only .80²=0.64. This severely reduces statistical power. One can solve this in theory using latent variable modeling. Sadly, the most commonly used framework for this in R (lavaan) does still not support interactions between latent terms, but there are now some other packages that allow for this (modsem). Just for curiosity, I refit the models above using this framework to see how much things change. This was one of those times when the last 5% of a study opened up a can of worms that would potentially take hours to resolve. The various versions of latent interaction models took a long time to fit (even for a single model), and/or produced unstable and implausible results, apparently related to inability to handle ordinal indicators (as opposed to continuous ones as normally used). Failing this, I fell back on a simpler method and brought out the trusty simex package. This is a clever method where instead of trying to model the error-free variables, you add more error to them and work backwards to 0 error. It only needs to know the amount of (random, classical) error, which is the same as the reliabilities. And it fit instantly. It tells us that in the above cases of the four models, coefficients are about 30% larger, so the variance they explain roughly 1.5-2x more variance, which means they are still quite small in relative terms, especially since the main effects also grow. Fitting the simex for all the scale-level models lets us generalize the findings further. These results showed that when taking into account measurement error, the interaction effects explained 1.3-2.5x as much variance, which in absolute terms was still too small to matter in most cases.

What does this mean for research? It means you should be wary of findings claiming interaction effects. Indeed, it was reported in one two of the large-scale replication studies that interaction effects replicated more poorly than average. In other words, since they are almost never large enough to matter, you should have a higher standard of evidence requirement to believe them. A p-value of < 5% is certainly not enough, and isn’t even enough for regular main effects. The lack of interaction effects also implies, indirectly, that selective/convenience samples are not likely to be severely biased. Why? Because whatever characteristics they are selected on are not likely to change the association you are interested in. The OKCupid dataset is a good demonstration of this. It replicated basically every normal finding I examined: 1) anti-social behavior and intelligence, 2) conservatism and pronatalism/fertility, 3) mental health and non-heterosexuality/trans, 4) astrology being bunk, 5) differences between dog and cat enthusiasts, and others. This is despite being an online convenience sample of strange people filling out 100s of questions on a dating site in semi-public.

Despite being based on a large international dataset, the dimensions are in fact not ‘supported by’ the data as such (in the exploratory factor analysis sense). They are just arbitrarily defined as the 2 factors from a curated list of items. You can read about that on the website for WVS:

The two dimensions have been created by running factor analysis over a set of ten indicators. The ten indicators used (five to tap each dimension) were chosen for technical reasons: in order to be able to compare findings across time, we used indicators that had been included in all four waves of the Values Surveys. These ten indicators reflect only a handful of the many beliefs and values that these two dimensions tap, and they are not necessarily the most sensitive indicators of these dimensions. They do a good job of tapping two extremely important dimensions of cross-cultural variation, but we should bear in mind that these specific items are only indicators of much broader underlying dimensions of cross-cultural variation [Source: Chapter 2 from Inglehart, R & C. Welzel. 2005. Modernization, Cultural Change and Democracy: The Human Development Sequence. New York: Cambridge University Press].

Since these data are publicly available, I decided to download them and try again using a theory-neutral approach. The WVS data has to be merged with the European Values Sutdy (EVS) to obtain data from the list of countries above (some European countries did not participate in WVS). After filtering to the common pool of items, we are left with only 44 common value items (asked in every country). Some items of high relevance are unfortunately missing, those dealing with homosexuality, because a few Muslim countries don’t allow researchers to ask these in surveys (shame on Egypt, Tajikistan, Uzbekistan, Iran). Moving forward with the 44 items with complete coverage for 92 countries (156,658 people), we can plot our trusty parallel analysis to get an idea of how many factors we could maximally extract:

So depending on whether we analyze averages by country (so 92 units) or persons (100k+) we get either a maximum of 4 or 13 factors. Due to aggregation issues, there is no guarantee these dimensions are even the same. It is not so uncommon that directions of association differ between levels of analysis. For instance, within a country, obesity correlates slightly negatively with intelligence (presumably becoming weaker because of GLP-1 drugs), but between countries the correlation is very strongly positive, as economic development is caused by intelligence and also makes food cheap and abundant to the point where many become obese (East Asians aside). As such, if one wanted to score a kind of social status general factor, then fitting it on individuals within countries would find a slightly negative factor loading and between countries would find a very strong positive loading. In general, though, levels of analysis agree and usually the country-level correlations are stronger and in the same direction as the person-level.

With regards to the above, I used the person-level data for more reliable results and because I wanted to score persons, not just countries. So how many factors to extract? I decided to retain factors which would be scored with a reliability of at least 0.70. This is a low standard, and the last factor is based on only 2 items, which is not great either. Honestly, one could have gone with the 5-factor solution as well, but I decided to include the 6th factor out of interest. Again, deciding on what factors to retain is not exact science, but depends on the goals of the researcher. In this case, I wanted a multi-dimensional scale that would produce interesting and at least somewhat reliable results. Looking at the resulting 6-factor solution, the dimensions were:

Amusingly, my own results suggest I am less sex egalitarian than the average Pakistani! Since I am not even religious let alone Muslim, it is hard to believe. The items that set me apart were:

I think the first one is fairly likely to be true given men’s extreme dominance under meritocratic hiring and even despite strong pro-female bias in the West. University is more important for boys mainly for the reason that women tend to study more useless stuff and for longer. Even female medical doctors commonly fail to make a return on investment on their medical degrees because “the median female physician simply doesn’t work enough hours to amortize her upfront investment in medical school”. American women fail to pay back their university tuition loans:

This extremist research has been popularized by the well-known far-right think tank The New York Times. One can also see that Blacks and Latinos should cut back on education spending.

The last question is a simple matter of family formation. If jobs are too scarce to support a dual income family, then let the man work, since he is likely to make more money, enjoy it more, and the women can focus on family matters. In a world of still falling sub-replacement fertility, this makes a lot of sense. Indeed, we already know that despite what Westerners say on the survey, they make the same decision when taking paid parental leave, and politicians have been forcing the men to take some of it in several countries to the detriment of the families. Anyway, the question talked about a right, which sounds like a legal right, and I am not keen on such an approach so I ended up picking “Neither”, which was still extreme by Scandinavian standards. As such, my responses are in line with the science and common sense, even if they ignore the gist of what the questions are getting at. If they had instead just asked people whether women should be banned from university education etc., then I would appear like a normal Westerner. Specific item choices can be important for results!

What about the famous 2 dimension plot that the World Values Survey people produced? Can it be recovered from a theory-neutral approach? Well, sort of. In my analysis, I let the dimensions covary, as value dimensions may do. Why does this matter? Well, when plotting the 2-dimension map of countries, it is somewhat of an implicit assumption of the plot that the dimensions sitting on X and Y are not themselves correlated. This is not really true unless we force them to be uncorrelated (orthogonal rotation as in PCA). The value data is similar to the intelligence data in that if we fit oblique rotations (potentially correlated factors), then the general traditionalism-modernism (left-right in slight disguise) dimension gets split up into its components, and the general factor will appear as a 2nd-order factor instead. We can avoid this by using an orthogonal rotation, in this case just PCA for simplicity. Doing this gives us these maps:

Scandinavians are at the top of the secular morality dimension with Sweden in a clear number 1, while the African-Islamic countries are on the other end. This won’t surprise you. Dimension 2 concerns corruption and democracy. Scandinavians are high again, but so are some Africans. The latter is odd given the prevalence of corruption there. Or maybe it is not so strange. After all, if corruption is prevalent, presumably a lot of citizens develop a dislike of it, even if they would also do it themselves if they could. Dog eat dog world. The other dimensions are somewhat harder to interpret without help so here’s a table:

Honestly, the other dimensions don’t always make sense. PC4 is a kind of my-local-people vs. globalism dimension.

I tried a variety of other approaches, but nothing I tried could make a neat, wholly scientifically satisfactory 2-dimensional plot. So what I ended up showing on the site is the 2nd-order modern values scale (from the 3 first factors, .99 correlated with PC1 as well), and interpersonal trust on the second dimension:

In terms of best fit for a country, the question is not really which country you sit closest to on a 2-dimensional map (whatever version of it we use), but how you place in all the 44-dimensions relative to country means. This 44-dimensional space is not something we can mentally visualize, but computing your position relative to countries is a trivial mathematical exercise. It turns out the country for me is Hong Kong:

However, we can see that the distances are somewhat close. I’ve never been to HK, Macao, Belarus, Vietnam etc., so it would seem a strange match. Amusingly, Denmark is one of the most distant matches to me! Clearly, this is not true if one used more sensible measures.

Finally, at the bottom, there are 2 metrics for how unusual your responses are, taken together. The first one is how unusual your responses were relative to what the 6-factor model predicts (mean absolute residual of responses predicted from your 6 factor scores). The second one is based on how far you are from the nearest country compared with other users. These are related metrics, but the first assumes a particular structure of the dataset, the second does not. They correlate about .70 in the norm data.

If you are familiar with personality psychology you know that there are endless debates about how many and which dimensions to rank people on. The Big Five (OCEAN)/5-factor model has been winning this battle for some decades, but before that it was Eysenck’s 3-factor predecessor, and before that Cattell’s 16 factor. Eysenck’s model has been mostly incorporated into the 5-factor model (except for psychoticism). Instead now we have the 6-factor HEXACO model, the alternative 5-factor model, as well as various simplifications, the big 2 (E+O, C+A+N), and the general factor of personality (GFP). However, I wanted to redo this work using the OKCupid data. This presented some interesting issues:

1. There is a lot of missing data because not everybody filled out every question, and some users specifically skipped some questions, or answered them in private (you could hide responses from others, so the scraper would not get them either).
2. The data structure is mixed. Many items are binary, some are ordinal (the options can be ranked from most to least something), and some are nominal (can’t be ordered). Analyzing mixed items is possible for factor analysis but requires specialized item response theory (IRT) models.
3. The dataset is large, so it took a very long time to fit the models, or impute the data. Like over a week even on a strong computer.

I had done some of this work years ago, and found the first few dimensions, but it was taking too long and I wasn’t happy with my methods. Fortunately, with the power of AI, I was able to revisit this question and get more decisive answers. Claude was able to rewrite the IRT software to use GPUs, which greatly sped up the model fitting time (by like 50x in core-time comparison but I have 16 CPU cores, so on average GPU was about 3x faster assuming full use), to the point where it was feasible to try out many models.

Generally speaking, we wish to know how many dimensions to extract. The standard answer in psychology is to use the various factor estimation methods with parallel analysis being the preferred option. The thing is that this always gives you a very large number when you have many cases because it is testing the actual seen structure vs. randomly generated responses. Just because parallel analysis or some other method says you should extract X factors (or the data supports it), doesn’t mean these are meaningful or reliably measured in that dataset. They may consist of just 2-3 items out of 100s. Since I wanted to make a practically useful test to put on the site, I wanted each dimension to be relatively reliable (at least .80) and rely on more than a few items. To figure this out, I explored 2 parameters: 1) the number of factors to extract (k), and 2) the number of items to use (p, in order of those with the most data in the database). Trying out the combinations of these values took a while (like 2 weeks on my strong computer even after changing to GPU fitting), but I am ready to present the results in brief. I guess I will eventually write up the full results, but building stuff with AI is much more interesting than writing academic papers, so I’ve been slacking off in that department (unlike the academics, I don’t have much financial motivation to write papers). So without further ado, these are the results for the estimated reliability of the least reliable factor as a function of those 2 parameters:

Alternatively, one can think of how adding another factor impacts the reliability metrics at a given item count:

Thus, in these data, the breakpoint is k=10 factors which is what I used for the scale on the site.

However, one can go further with the optimization. One doesn’t have to use the first p items from the site, but one can explore a larger number of items and pick an optimal subset. Thus, it becomes a variable selection problem after the exploration stage. Using the same criterion as above, I found that one needs about 200 items to reach .80 reliability for the 10 factors of interest. This approach explicitly uses cross-loadings to boost reliability. Most personality scales avoid items with cross-loadings because the scales were (and are) optimized using a simple metric of internal consistency, Cronbach’s alpha. This metric becomes 1 when every question is essentially a duplicate, which is why so many scales have many duplicate questions. This is very inefficient from an information gathering perspective. One would prefer quite dissimilar items that measure multiple aspects of personality at once. (I got this perspective from this blogpost.) This is what the OKCupid scale implements. My results look like this:

Some somewhat amusing results. This is largely because the scales are impure. There are multiple questions about guns — this being an American test — which tend to go with edginess. I’ve never even shot a gun, but still got a high score due to a fondness for inappropriate humor. Here are the top 5 most important questions for each dimension:

Casual-sex orientation:

Fitness & vitality:

2 questions are actually about transsexuals.

Sexual experience:

One question is actually about the user’s own fertility and another about potential partner’s fertility.

Guns and edginess:

Notice how 2 questions are actually about race and eugenics, not guns or crude humor.

Religiosity:

Social liberalism:

Yet another question about race.

Partner pickiness:

Some of these questions are quite sex-specific.

Intellectualism:

Questions concern sexual prudishness, self-rated intelligence, flag burning and so on. These items go together statistically speaking even if they measure seemingly independent things.

Kink and sexual exploration:

Substance use:

Overall we can say that these dimensions are somewhat unsatisfactory because they mix up different things. From this perspective, we should want to have more and more narrow dimensions. However, this conflicts with the goal of having reliable scales. A trade-off must be made here. You may have different standards than me. Factor analysis and scale construction remain somewhat of an art and not purely a scientific matter. The above is my best current attempt at getting useful dimensions out of the OKCupid data. Maybe in the future I will make another attempt. For now, let’s hope these dimensions are measuring important things and we will discover their correlations. If anything, they provide us with 200 questions from a larger pool, which could be analyzed in different ways for research purposes.

1. We don’t have many people who took all the tests.
2. We have severe sample selection bias, takers are far above average, and some tests have severe ceiling effects.
3. Tests vary in reliability, some are short, some long.
4. Tests vary in English ability dependency. Non-natives are not likely to perform at their true verbal ability level on a foreign language test, especially not vocabulary.

We can attempt to get around these problems by:

1. Using pairwise complete cases to estimate correlations among each pair of tests. From these we can build a reasonably full correlation matrix, which allows us to use matrix-based methods including factor analysis/structural equation modeling.
2. We can estimate sample selection bias, and attempt to adjust for ceiling bias.
3. We can estimate test reliabilities or use reference values and adjust for these.
4. We don’t have data on what language people speak natively, but we have IPs and thus can infer countries with some accuracy. Then we can assume that people in English speaking countries are mostly native speakers and those elsewhere mostly not. We can compare results.

Starting with (4), we can have a brief look if it even looks like we have to do anything:

So they are somewhat different, but not terribly so. The traditional metric for similarity of loadings is factor congruence which is 0.99. The one oddity is ICAR16 in full data vs. subsets. The reason for that is that ICAR60 has too little data for the subsets, so it is excluded and this artificially changes the g-loading since ICAR16 is a subset of ICAR60. Ideally, one would not include partial duplicates, but we can’t be too picky here. So maybe we don’t need to do anything for the correlations. The means, however, are quite different:

This is the familiar language and cultural bias, which is of course the largest on tests of such things, US knowledge tests. Curiously, the supposedly fluid intelligence test in UKBB shows a large bias. It’s a short time-limited test, so it indirectly measures English reading speed. The language-minimal tests (figures, ICAR, mental rotation) show quite small gaps as expected (except for number series, but it’s a tiny sample). We also get the expected finding that non-English speaking West knows more languages. This is also trivially true in the sense that English wasn’t one of the languages tested.

Regarding the pairwise complete correlations, they look like this:

These are the values used to do the factor analysis to get the g-loadings above. There is a lot of holes in this matrix, which one can either impute, or drop that particular test. A choice has to be made about how much to impute versus drop. A matter of researcher taste to some extent. The long vocabulary test with Prolific norms only has 4 correlations observed so far (it was posted yesterday), and it would be unwise to make much of those, though it does have a 0.61 correlation with Wordsum.

Hypothetically, the question we are asking is: in a dream world where all the tests are infinitely long and thus perfectly reliable (in classical test theory sense), and where all the test subjects are perfectly representative in terms of intelligence, and where every test has infinite range, what are the correlations among the tests? These are the values we need to get the true g-loadings. So since we have issues, we have to correct. Ceiling effects have a variety of existing solutions. Just for reference, this is what our Wordsum vs. American general population data look like:

25% of our online subjects got a perfect score versus 3% in the US White general population. It is easy to imagine how the correlation must decrease the bigger the lumping of scores at one end, since people of dissimilar ability are obtain the same numerical score. One simple solution I like is to bin the values and use the latent correlation from ordinal-ordinal or ordinal-continuous functions (polychoric/polyserial correlation). This can also be used to infer the split-half reliability estimate without bias from the ceiling, though it is still biased from the selection. Thus, to obtain an estimate of our hypothetically perfect correlation matrix we do these steps:

1. Ceiling bias correction via binning + polychoric.
2. Reliability correction (using value from split-half polychoric).
3. Selection correction

The latter is problematic to estimate. The value the statistical correction needs is the SD in our sample vs. population. However, the SD is affected by ceiling bias, which results in a far too small SD. This would mean the correction is too strong and correlations too high. Unfortunately, the only tests we have with truly representative samples for are the Wordsum and the Pew Research science, both which have severe ceiling issues for our subjects. We have data from the SAPA project for ICAR. The 16-item version is also affected by ceiling bias (15% in our data vs. 2% in SAPA’s). The people who took the time to take the full 60 item version are even more selected. I tried a variety of methods and they produced selection ratios of 0.50 to even above 1. I don’t know what the selection ratio really is, but I’ve used 0.70 in my calculations until we can find something better. This is what we get:

We have a number of too weak correlations, but that is in part because the raw correlations were close to 0 by sampling error. Corrections cannot fix this issue, only more data can. In any case, every correlation except for one (-0.05) is positive despite the sampling error. So finally, we get the answer for g-loadings we wanted:

Well, sort of. Some of the values are clearly wrong. ICAR60 is a long 4-part test, the true g-loading will surely be 0.90 or so, yet we found 0.54. Treat the above as minimum values more than unbiased estimates. These estimates are based on very thin overlaps due to small samples. As more data come in, we will be able to update these values. The most g-loaded tests are the usual quick knowledge tests. This is annoying because these are the most culturally sensitive, but on the other hand, they are fast and unstressful. This collection of tests also has too many knowledge tests, so the g factor will be somewhat colored, boosting their values slightly. As we accumulate more non-verbal tests, we will be able to fit more complex models to account for this as is typically done. We should then eventually recover the usual finding that fluid ability is the same thing as g on a theoretical level, though not using our observed scores.

Extras

The evolution of the correlation matrix for those curious:

First off, as AI wrote in the test description:

Political opinion is usually summarised as one line from left to right, and that line is real: Americans’ views on guns, welfare, abortion, race, immigration and defense all move together. But they do not move in lockstep. A factor analysis of 83 policy questions from the 2024 American National Election Study finds eight distinguishable dimensions, six of them strongly tied to the left–right axis and two — distrust of government and the wish to hand decisions to experts, business leaders or referendums — largely independent of it.

This test asks every ANES question that belongs clearly to one of the eight dimensions — 73 in all, in the ANES’s own wording — and scores you against the ANES respondents, weighted to represent US adults. A question that bears on two dimensions counts toward both. For every dimension and for the overall left–right score you see where you fall among all Americans and among self-identified Democrats, independents and Republicans, with the three parties’ distributions drawn on top of each other. The overlap is the point: on most single issues, a fair share of Republicans sit left of the median Democrat and vice versa.

The overall left–right score is a single factor fitted through all 73 questions: the one dimension that runs through the eight, scored directly from the items. In the ANES it correlates 0.80 with respondents’ own liberal–conservative self-placement and 0.77 with party identification. The questions are American; non-Americans can take it, but the norms describe where Americans stand.

So we begin with the 2024 wave of the ANES survey. It had 83 policy questions that we could factor analyze. Now, the thorny question in the years of latent variable modeling is how to determine the factor structure of some dataset. There are many methods for this: Kaiser’s criterion (eigenvalue > 1), MAP, model fit indexes and so on. The most popular is the parallel analysis. In this method, we simulate random but similar data at the same sample size and check their distribution of eigenvalues. We repeat this a number of times to get a distribution. Then we extract and keep all factors that exceed that value. For a very large dataset such as ours, this would mean keeping a large number of factors. First, the dataset correlations look like this:

Items are turned so that higher is the right-wing direction if any. This is because otherwise it would be more difficult to discern the overwhelming fact of such a dataset, namely, that most but not all questions load on a general factor. In intelligence data, this would be g, in political science, it is the left-right axis, which we would call leftism or conservatism depending on the direction and chosen framing. Applying parallel analysis to the ANES data we get this:

So parallel analysis tells us that the dataset can support about 15 factors. Model fit via RMSEA also shows that adding more factors does better. However, for practical purposes, we may not want just about any factor that can be seen. Even if they can be extracted, it does not mean they are meaningful or stable. So in my case, I decided to also require factors to be based on at least 4 items with salient loadings of >0.30. If not enforced, one can easily get factors that result from just 2 questions, which may just be oppositely worded repeats. One could have done this analysis differently and ended up with more or fewer factors. For instance, in the k=10 solution, race & gender grievances split into their race and gender parts, which may be important in some edge cases. In the appendix, there’s a bass-ackwards plot for those very interested. Anyway, based on my preferred 8-factor model, 74 items had a salient loading, 3 did not and were dropped, and 1 more item was dropped because of missing data (it was a near duplicate anyway). Of the remaining 73 items, they load like this:

These factors are themselves somewhat correlated, so one can factor analyze them again to get the second order general factor as in hierarchical factor analysis in intelligence. Their factor correlations are:

We can see that the 8 dimensions are correlated positively, though the last 2 much less so. I decided to score the general left-right axis from the 73 questions directly. This unconstrains the path from the general factor to the items to be more free. By looking at the plot above you can see that this makes little difference as the hierarchical left-right dimension correlates 0.99 with the one extracted directly.

Using this model, we can check the distribution of scores in ANES and from our test takers online:

Online takers are more bimodal but only for the 6 scales that have strong left-right loadings, and the last 2 are more unimodal, though not necessarily normally distributed. Finally, then, we can compare the left-right dimension the same way:

This is all well and good, but do these factor scores for left-right work? Well, I left out the self-placement items, so they could be used for validation:

People’s own ratings of their ideology and party allegiance correlate about 0.80 with the score. For the parties, apparently X and leans X are about the same thing. The gap sizes in these policy opinions are quite large:

As a function of age and sex, results are eye-opening:

In the US nationally representative data, men are more right-wing at a roughly constant effect size of 0.3 SD at all ages, and older people are more right-wing. In the online data, this pattern is extremely different before the age of 35. If one was only looking at online data, it would appear that young women have taken an extreme left-wing turn. I am not quite sure which plot is more realistic. The online sample is obviously self-selected but ANES-type data also suffer from low data quality as the takers are not very motivated. To note is that I only added the sex and age questions later so the full 28k dataset does not have this information, which may create further self-selection issues.

Appendix

Here’s a bass-ackwards type plot for the k=1-15 solutions with tentative labels (AI):

The p is the number of items with >.30 loadings. Some people were insinuating that I had deliberately merged nationalism with Israel love in some kind of neo-con plot. These people would perhaps prefer the k=9 solution where “Hawkish nationalism” splits the 10 items into global interventionism (Israel, Ukraine stuff) vs. domestic (border wall, voter ID etc.). This solution was not chosen because the technocrat factor declines from 4 to 3 salient loadings. If one added more or different questions to the dataset, one would of course get somewhat different factors. Factor analysis cannot find factors in a dataset that isn’t suitable for finding them.

• Arden, R., Gottfredson, L. S., Miller, G., & Pierce, A. (2009). Intelligence and semen quality are positively correlated. Intelligence, 37(3), 277-282.
• Pierce, A., Miller, G., Arden, R., & Gottfredson, L. S. (2009). Why is intelligence correlated with semen quality? Biochemical pathways common to sperm and neuron function and their vulnerability to pleiotropic mutations: Biochemical pathways common to sperm and neuron function and their vulnerability to pleiotropic mutations. Communicative & Integrative Biology, 2(5), 385-387.

Using the Vietnam Experience Study (US military veterans aged ca. 38), one of the only datasets that ever measured these two phenotypes together, they found correlations like these:

As far as I can tell, there is only one replication of this and it’s n=41 and didn’t find anything.

Since I have this dataset, I decided to look again. Arden et al did not show the scatterplot, but they should have because it looks like this:

This is the full sample, no adjustments or exclusions aside from missing data. The correlation is there, 0.14, vs. their 0.19, but if you look at the deciles on the right, you can see that the signal is entirely in the bottom 10% of men. This is more or less the case for every sperm metric that shows any signal:

Some of these metrics are interdependent since normal + small + large + tapered + amorphous = 100%. Thus, the simplest way to analyze that is just to use normal%.

So is this a real thing? Maybe the bottom 5-10% of men really do have quite a bit worse semen quality, or maybe there is some problem. Perhaps they were drunk at or before the testing, so their cognitive scores are bad and so is their semen quality. Or perhaps they didn’t stick to the rule of abstinence before ejaculation day. But before that, let’s examine the semen measures + covariates:

We can see moderate correlations among the semen metrics in an expected way. Factor analysis also confirms a 2-factor solution for volume and quality, so I will use those going forward. For good measure, here are the decile plots for semen quality and volume:

After looking closely at the 5% bins (not shown here), there is only an apparent effect in the bottom 10% of men sorted by g (intelligence), so I will use a binary for this group in the models. Here are some model results:

The results are pretty interesting. The bottom 10% men by intelligence have about 0.85 standard deviations worse semen quality, but only 0.35 SD less volume (p < 5% only). The quality link does not go away even with extensive controls, while the volume link became weaker and non-significant. The bottom 10% is only 48 men, so there is a limit to how much statistical precision we have to work with here, and we can’t say for sure about the volume. Note that Blacks have less semen volume than Whites, as may be expected by their smaller testicles — a finding we previously reported from this dataset. Nothing predicts semen quality as well as low intelligence, though strangely abstinence predicts it mildly, perhaps not a biological link but a behavioral. A crude count of medical history (lifetime diagnosed conditions) predicts it slightly perhaps, while smoking, drinking, and BMI had no detectable effect. Concerning volume, abstinence is a decent predictor.

So it seems somewhat safe to say that intelligence does predict semen quality, but apparently only because the bottom 10% have really terrible quality. This is less easy to explain than positing some general fitness factor that underlies both, as done originally.