Blog

Betas and residualized variables / does non-g ability predict GPA?

https://twitter.com/pnin1957/status/641284556818096128

Let’s test that. Since we don’t have the original data, we can’t use that. We can however use open datasets. I like to use Wicherts’ dataset. So let’s analyze!

library(pacman)
p_load(kirkegaard, psych, stringr)

#load wicherts data
w = read.csv("wicherts_data.csv", sep = ";")
w = subset(w, select = c("ravenscore", "lre_cor", "nse_cor", "voc_cor", "van_cor", "hfi_cor", "ari_cor", "Zgpa"))

#remove missing
w = na.omit(w)

#standardize
w = std_df(w)

#subset the subtests
w_subtests = subset(w, select = c("ravenscore", "lre_cor", "nse_cor", "voc_cor", "van_cor", "hfi_cor", "ari_cor"))

#factor analyze
fa = fa(w_subtests)

#plot
plot_loadings(fa) + scale_x_continuous(breaks = seq(0, 1, .05))

#save scores
w_subtests$g = as.numeric(fa$scores)

#residualize
w_res = residualize_DF(w_subtests, "g")

#include GPA
w_res$GPA = w$Zgpa

#cors
write_clipboard(wtd.cors(w_res))

#predict GPA
fits = lm_beta_matrix(dependent = "GPA",
                      predictors = colnames(w_res)[-9],
                      data = w_res, return_models = "n")

write_clipboard(fits)

#why does the last model have NA for one variable?
model = str_c("GPA ~ ", str_c(colnames(w_res)[-9], collapse = " + "))
fit = lm(model, w_res)
summary(fit)

Results

wicherts_factor_loadings

Correlations

ravenscore lre_cor nse_cor voc_cor van_cor hfi_cor ari_cor g GPA
ravenscore 1 -0.125 -0.201 -0.115 0.057 0.022 -0.294 0 -0.028
lre_cor -0.125 1 -0.316 -0.143 -0.195 -0.23 -0.288 0 -0.074
nse_cor -0.201 -0.316 1 -0.206 -0.381 -0.103 0.099 0 -0.116
voc_cor -0.115 -0.143 -0.206 1 0.198 -0.141 -0.207 0 0.08
van_cor 0.057 -0.195 -0.381 0.198 1 -0.081 -0.406 0 0.137
hfi_cor 0.022 -0.23 -0.103 -0.141 -0.081 1 -0.233 0 0.037
ari_cor -0.294 -0.288 0.099 -0.207 -0.406 -0.233 1 0 0.007
g 0 0 0 0 0 0 0 1 0.334
GPA -0.028 -0.074 -0.116 0.08 0.137 0.037 0.007 0.334 1

 

Model fits

Model # ravenscore lre_cor nse_cor voc_cor van_cor hfi_cor ari_cor g r2.adj.
1 -0.028               -0.003
2 -0.074 0.002
3 -0.116 0.01
4 0.08 0.003
5 0.137 0.015
6 0.037 -0.002
7 0.007 -0.003
8 0.334 0.108
9 -0.038 -0.079 0
10 -0.054 -0.127 0.009
11 -0.019 0.078 0
12 -0.036 0.139 0.013
13 -0.029 0.038 -0.005
14 -0.029 -0.001 -0.006
15 -0.028 0.334 0.106
16 -0.123 -0.155 0.02
17 -0.064 0.071 0.004
18 -0.049 0.127 0.014
19 -0.069 0.021 -0.001
20 -0.078 -0.015 -0.001
21 -0.074 0.334 0.111
22 -0.104 0.059 0.01
23 -0.075 0.108 0.017
24 -0.114 0.026 0.007
25 -0.118 0.019 0.007
26 -0.116 0.334 0.119
27 0.055 0.126 0.015
28 0.087 0.05 0.002
29 0.086 0.025 0
30 0.08 0.334 0.112
31 0.141 0.049 0.014
32 0.168 0.075 0.017
33 0.137 0.334 0.124
34 0.041 0.017 -0.005
35 0.037 0.334 0.107
36 0.007 0.334 0.105
37 -0.082 -0.14 -0.177 0.023
38 -0.029 -0.068 0.067 0.001
39 -0.043 -0.054 0.129 0.013
40 -0.038 -0.074 0.021 -0.003
41 -0.049 -0.09 -0.033 -0.003
42 -0.038 -0.079 0.334 0.109
43 -0.046 -0.115 0.052 0.008
44 -0.052 -0.086 0.107 0.016
45 -0.054 -0.125 0.026 0.007
46 -0.053 -0.127 0.004 0.006
47 -0.054 -0.127 0.334 0.119
48 -0.03 0.052 0.128 0.012
49 -0.02 0.085 0.05 -0.001
50 -0.013 0.083 0.021 -0.003
51 -0.019 0.078 0.334 0.109
52 -0.038 0.143 0.05 0.012
53 -0.017 0.166 0.07 0.014
54 -0.036 0.139 0.334 0.122
55 -0.027 0.04 0.009 -0.008
56 -0.029 0.038 0.334 0.104
57 -0.029 -0.001 0.334 0.103
58 -0.115 -0.146 0.034 0.018
59 -0.097 -0.119 0.072 0.021
60 -0.125 -0.157 -0.008 0.017
61 -0.127 -0.155 -0.014 0.017
62 -0.123 -0.155 0.334 0.129
63 -0.043 0.051 0.118 0.013
64 -0.055 0.078 0.036 0.001
65 -0.062 0.072 0.004 0
66 -0.064 0.071 0.334 0.113
67 -0.039 0.133 0.039 0.012
68 -0.024 0.158 0.065 0.014
69 -0.049 0.127 0.334 0.123
70 -0.072 0.018 -0.009 -0.005
71 -0.069 0.021 0.334 0.108
72 -0.078 -0.015 0.334 0.108
73 -0.068 0.046 0.102 0.015
74 -0.099 0.065 0.036 0.008
75 -0.106 0.065 0.031 0.007
76 -0.104 0.059 0.334 0.119
77 -0.069 0.114 0.039 0.015
78 -0.07 0.139 0.071 0.017
79 -0.075 0.108 0.334 0.126
80 -0.116 0.031 0.026 0.004
81 -0.114 0.026 0.334 0.116
82 -0.118 0.019 0.334 0.116
83 0.063 0.129 0.057 0.015
84 0.067 0.158 0.085 0.017
85 0.055 0.126 0.334 0.124
86 0.098 0.061 0.042 0
87 0.087 0.05 0.334 0.111
88 0.086 0.025 0.334 0.109
89 0.183 0.075 0.099 0.018
90 0.141 0.049 0.334 0.123
91 0.168 0.075 0.334 0.126
92 0.041 0.017 0.334 0.104
93 -0.078 -0.135 -0.171 0.017 0.02
94 -0.076 -0.116 -0.144 0.064 0.023
95 -0.082 -0.144 -0.179 -0.012 0.02
96 -0.099 -0.158 -0.181 -0.05 0.022
97 -0.082 -0.14 -0.177 0.334 0.133
98 -0.036 -0.048 0.045 0.121 0.011
99 -0.028 -0.059 0.074 0.035 -0.002
100 -0.033 -0.072 0.064 -0.01 -0.003
101 -0.029 -0.068 0.067 0.334 0.11
102 -0.042 -0.044 0.134 0.039 0.011
103 -0.026 -0.032 0.154 0.053 0.011
104 -0.043 -0.054 0.129 0.334 0.122
105 -0.048 -0.085 0.012 -0.029 -0.006
106 -0.038 -0.074 0.021 0.334 0.106
107 -0.049 -0.09 -0.033 0.334 0.107
108 -0.046 -0.079 0.039 0.102 0.014
109 -0.045 -0.11 0.058 0.035 0.006
110 -0.04 -0.115 0.056 0.018 0.005
111 -0.046 -0.115 0.052 0.334 0.118
112 -0.052 -0.08 0.113 0.039 0.014
113 -0.034 -0.078 0.133 0.059 0.015
114 -0.052 -0.086 0.107 0.334 0.125
115 -0.051 -0.125 0.028 0.011 0.003
116 -0.054 -0.125 0.026 0.334 0.116
117 -0.053 -0.127 0.004 0.334 0.115
118 -0.03 0.059 0.132 0.057 0.012
119 -0.005 0.066 0.158 0.084 0.014
120 -0.03 0.052 0.128 0.334 0.122
121 -0.007 0.096 0.06 0.039 -0.003
122 -0.02 0.085 0.05 0.334 0.108
123 -0.013 0.083 0.021 0.334 0.106
124 -0.013 0.182 0.074 0.095 0.015
125 -0.038 0.143 0.05 0.334 0.122
126 -0.017 0.166 0.07 0.334 0.123
127 -0.027 0.04 0.009 0.334 0.101
128 -0.091 -0.112 0.03 0.071 0.018
129 -0.115 -0.145 0.034 0.001 0.014
130 -0.117 -0.146 0.033 -0.005 0.014
131 -0.115 -0.146 0.034 0.334 0.127
132 -0.094 -0.116 0.075 0.01 0.017
133 -0.081 -0.11 0.092 0.032 0.018
134 -0.097 -0.119 0.072 0.334 0.13
135 -0.132 -0.158 -0.014 -0.019 0.014
136 -0.125 -0.157 -0.008 0.334 0.126
137 -0.127 -0.155 -0.014 0.334 0.127
138 -0.03 0.059 0.123 0.049 0.012
139 -0.011 0.065 0.155 0.08 0.014
140 -0.043 0.051 0.118 0.334 0.123
141 -0.045 0.085 0.044 0.022 -0.002
142 -0.055 0.078 0.036 0.334 0.111
143 -0.062 0.072 0.004 0.334 0.11
144 0.012 0.189 0.08 0.106 0.015
145 -0.039 0.133 0.039 0.334 0.122
146 -0.024 0.158 0.065 0.334 0.123
147 -0.072 0.018 -0.009 0.334 0.105
148 -0.059 0.054 0.108 0.047 0.014
149 -0.061 0.058 0.135 0.08 0.017
150 -0.068 0.046 0.102 0.334 0.125
151 -0.1 0.076 0.048 0.044 0.006
152 -0.099 0.065 0.036 0.334 0.117
153 -0.106 0.065 0.031 0.334 0.117
154 -0.059 0.157 0.065 0.092 0.018
155 -0.069 0.114 0.039 0.334 0.124
156 -0.07 0.139 0.071 0.334 0.127
157 -0.116 0.031 0.026 0.334 0.114
158 0.083 0.175 0.09 0.117 0.021
159 0.063 0.129 0.057 0.334 0.124
160 0.067 0.158 0.085 0.334 0.127
161 0.098 0.061 0.042 0.334 0.109
162 0.183 0.075 0.099 0.334 0.128
163 -0.072 -0.112 -0.139 0.015 0.063 0.019
164 -0.079 -0.138 -0.174 0.015 -0.009 0.016
165 -0.1 -0.158 -0.182 -0.002 -0.05 0.018
166 -0.078 -0.135 -0.171 0.017 0.334 0.13
167 -0.075 -0.115 -0.143 0.065 0.003 0.019
168 -0.083 -0.127 -0.152 0.052 -0.018 0.019
169 -0.076 -0.116 -0.144 0.064 0.334 0.133
170 -0.106 -0.173 -0.19 -0.034 -0.063 0.019
171 -0.082 -0.144 -0.179 -0.012 0.334 0.13
172 -0.099 -0.158 -0.181 -0.05 0.334 0.132
173 -0.035 -0.035 0.053 0.125 0.048 0.01
174 -0.01 -0.015 0.063 0.153 0.075 0.011
175 -0.036 -0.048 0.045 0.121 0.334 0.121
176 -0.024 -0.054 0.077 0.038 0.009 -0.005
177 -0.028 -0.059 0.074 0.035 0.334 0.108
178 -0.033 -0.072 0.064 -0.01 0.334 0.107
179 -0.01 0.008 0.186 0.078 0.1 0.012
180 -0.042 -0.044 0.134 0.039 0.334 0.12
181 -0.026 -0.032 0.154 0.053 0.334 0.121
182 -0.048 -0.085 0.012 -0.029 0.334 0.104
183 -0.044 -0.07 0.046 0.107 0.046 0.012
184 -0.022 -0.067 0.053 0.131 0.072 0.014
185 -0.046 -0.079 0.039 0.102 0.334 0.124
186 -0.034 -0.108 0.067 0.044 0.032 0.003
187 -0.045 -0.11 0.058 0.035 0.334 0.116
188 -0.04 -0.115 0.056 0.018 0.334 0.115
189 -0.028 -0.066 0.152 0.062 0.082 0.015
190 -0.052 -0.08 0.113 0.039 0.334 0.124
191 -0.034 -0.078 0.133 0.059 0.334 0.125
192 -0.051 -0.125 0.028 0.011 0.334 0.113
193 0.004 0.083 0.176 0.091 0.118 0.018
194 -0.03 0.059 0.132 0.057 0.334 0.122
195 -0.005 0.066 0.158 0.084 0.334 0.124
196 -0.007 0.096 0.06 0.039 0.334 0.106
197 -0.013 0.182 0.074 0.095 0.334 0.125
198 -0.083 -0.105 0.035 0.076 0.018 0.015
199 -0.065 -0.095 0.042 0.098 0.046 0.016
200 -0.091 -0.112 0.03 0.071 0.334 0.128
201 -0.118 -0.146 0.032 -0.002 -0.006 0.011
202 -0.115 -0.145 0.034 0.001 0.334 0.124
203 -0.117 -0.146 0.033 -0.005 0.334 0.124
204 -0.053 -0.089 0.12 0.039 0.059 0.015
205 -0.094 -0.116 0.075 0.01 0.334 0.127
206 -0.081 -0.11 0.092 0.032 0.334 0.128
207 -0.132 -0.158 -0.014 -0.019 0.334 0.124
208 0.044 0.094 0.195 0.11 0.144 0.019
209 -0.03 0.059 0.123 0.049 0.334 0.122
210 -0.011 0.065 0.155 0.08 0.334 0.124
211 -0.045 0.085 0.044 0.022 0.334 0.108
212 0.012 0.189 0.08 0.106 0.334 0.125
213 -0.044 0.075 0.157 0.082 0.11 0.019
214 -0.059 0.054 0.108 0.047 0.334 0.124
215 -0.061 0.058 0.135 0.08 0.334 0.127
216 -0.1 0.076 0.048 0.044 0.334 0.116
217 -0.059 0.157 0.065 0.092 0.334 0.128
218 0.083 0.175 0.09 0.117 0.334 0.131
219 -0.071 -0.109 -0.136 0.017 0.065 0.007 0.016
220 -0.078 -0.12 -0.145 0.011 0.056 -0.011 0.016
221 -0.072 -0.112 -0.139 0.015 0.063 0.334 0.13
222 -0.117 -0.188 -0.201 -0.024 -0.045 -0.077 0.016
223 -0.079 -0.138 -0.174 0.015 -0.009 0.334 0.127
224 -0.1 -0.158 -0.182 -0.002 -0.05 0.334 0.128
225 -0.09 -0.142 -0.163 0.038 -0.014 -0.032 0.016
226 -0.075 -0.115 -0.143 0.065 0.003 0.334 0.13
227 -0.083 -0.127 -0.152 0.052 -0.018 0.334 0.13
228 -0.106 -0.173 -0.19 -0.034 -0.063 0.334 0.129
229 0.023 0.056 0.101 0.201 0.118 0.16 0.016
230 -0.035 -0.035 0.053 0.125 0.048 0.334 0.12
231 -0.01 -0.015 0.063 0.153 0.075 0.334 0.121
232 -0.024 -0.054 0.077 0.038 0.009 0.334 0.105
233 -0.01 0.008 0.186 0.078 0.1 0.334 0.122
234 -0.009 -0.046 0.072 0.155 0.08 0.106 0.016
235 -0.044 -0.07 0.046 0.107 0.046 0.334 0.123
236 -0.022 -0.067 0.053 0.131 0.072 0.334 0.124
237 -0.034 -0.108 0.067 0.044 0.032 0.334 0.114
238 -0.028 -0.066 0.152 0.062 0.082 0.334 0.125
239 0.004 0.083 0.176 0.091 0.118 0.334 0.128
240 0.015 -0.034 0.08 0.168 0.09 0.121 0.016
241 -0.083 -0.105 0.035 0.076 0.018 0.334 0.125
242 -0.065 -0.095 0.042 0.098 0.046 0.334 0.126
243 -0.118 -0.146 0.032 -0.002 -0.006 0.334 0.121
244 -0.053 -0.089 0.12 0.039 0.059 0.334 0.126
245 0.044 0.094 0.195 0.11 0.144 0.334 0.129
246 -0.044 0.075 0.157 0.082 0.11 0.334 0.13
247 -0.071 -0.109 -0.136 0.017 0.065 0.007 0.016
248 -0.071 -0.109 -0.136 0.017 0.065 0.007 0.334 0.127
249 -0.078 -0.12 -0.145 0.011 0.056 -0.011 0.334 0.127
250 -0.117 -0.188 -0.201 -0.024 -0.045 -0.077 0.334 0.127
251 -0.09 -0.142 -0.163 0.038 -0.014 -0.032 0.334 0.127
252 0.023 0.056 0.101 0.201 0.118 0.16 0.334 0.127
253 -0.009 -0.046 0.072 0.155 0.08 0.106 0.334 0.127
254 0.015 -0.034 0.08 0.168 0.09 0.121 0.334 0.127
255 -0.071 -0.109 -0.136 0.017 0.065 0.007 0.334 0.127

[Bonus points to whoever can explain why the last ari_cor has a missing value in the last model. I checked. It is not a problem with my function. I don’t know.]

So Timofey Pnin is right. The beta does stay exactly the same across models, at least two 3 digits.

We may also note that adding the other predictors did not have much effect: g alone (model #8) R2 adj. = .108, best model according to R2 adj. = 0.132 (#97). Notice how this model has negative betas for the other items. In other words, one is better off with lower scores. Surely that can’t be right. It is probably just a fluke due to overfitting…

Testing overfitting

We can test overfitting using lasso regression (read this book, seriously, it’s a great book!). Because lasso regression is indeterministic, we repeat it a large number of times and examine the overall results.

#lasso regression
fits_2 = MOD_repeat_cv_glmnet(df = w_res,
                              dependent = "GPA",
                              predictors = colnames(w_res)[-9],
                              runs = 500)

write_clipboard(MOD_summarize_models(fits_2))

Lasso results

ravenscore lre_cor nse_cor voc_cor van_cor hfi_cor ari_cor g
mean 0 0 0 0 0 0 0 0.096
median 0 0 0 0 0 0 0 0.104
sd 0 0 0 0 0.001 0 0 0.033
fraction_zeroNA 1 1 1 1 0.996 1 1 0.01

 

The lasso confirms our suspicions. The non-g variables were fairly useless, their apparent usefulness due to overfitting. g retained its usefulness in 99% of the runs. The most promising of the other candidates was only useful in .04% of runs.

This is probably worth writing into a short paper. Contact me if you are willing to do this. I will help you, but I don’t have time to write it all myself.

Tags: , , , , ,