{"id":10761,"date":"2022-08-06T03:29:44","date_gmt":"2022-08-06T02:29:44","guid":{"rendered":"https:\/\/emilkirkegaard.dk\/en\/?p=10761"},"modified":"2022-08-06T08:05:48","modified_gmt":"2022-08-06T07:05:48","slug":"the-signal-and-the-noise-in-meta-analysis","status":"publish","type":"post","link":"https:\/\/emilkirkegaard.dk\/en\/2022\/08\/the-signal-and-the-noise-in-meta-analysis\/","title":{"rendered":"The signal and the noise in meta-analysis"},"content":{"rendered":"<p>OK, it has been <a href=\"https:\/\/stuartritchie.substack.com\/p\/pseudocritics\">written about already<\/a> by <a href=\"https:\/\/www.reddit.com\/r\/badeconomics\/comments\/we9r4r\/rsquared_as_a_measure_of_a_study_quality\/\">some others<\/a>, but I also want to talk about this John Protzko pile-on:<\/p>\n<div class=\"oceanwp-oembed-wrap clr\">\n<blockquote class=\"twitter-tweet\" data-width=\"550\" data-dnt=\"true\">\n<p lang=\"en\" dir=\"ltr\">About 3 weeks ago I thought: &quot;I wonder if behavior on economic games has been getting more selfish over time. But doing that study would be A LOT of work.&quot;<\/p>\n<p>Thankfully, someone went through and found out<\/p>\n<p>1956-2017, people have been getting less selfish<a href=\"https:\/\/t.co\/HXs71OTEkF\">https:\/\/t.co\/HXs71OTEkF<\/a> <a href=\"https:\/\/t.co\/bjjJvUVb7z\">pic.twitter.com\/bjjJvUVb7z<\/a><\/p>\n<p>&mdash; John Protzko (@JProtzko) <a href=\"https:\/\/twitter.com\/JProtzko\/status\/1554112968245137408?ref_src=twsrc%5Etfw\">August 1, 2022<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/div>\n<p>Here&#8217;s the original plot in case the tweet goes down:<\/p>\n<p><a href=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/protzko-orig.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-10762\" src=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/protzko-orig.png\" alt=\"\" width=\"545\" height=\"501\" srcset=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/protzko-orig.png 545w, https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/protzko-orig-300x276.png 300w\" sizes=\"auto, (max-width: 545px) 100vw, 545px\" \/><\/a><\/p>\n<p>First we might notice that the error bands are unusually large. Protzko didn&#8217;t actually make the plot, as some critics imply. Yuan et al did it themselves in their study:<\/p>\n<ul>\n<li>Yuan, M., Spadaro, G., Jin, S., Wu, J., Kou, Y., Van Lange, P. A., &amp; Balliet, D. (2022). <a href=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/Did-Cooperation-Among-Strangers-Decline-in-the-United-States-A-Cross-Temporal-Meta-Analysis-of-Social-Dilemmas-1956\u20132017.pdf\">Did cooperation among strangers decline in the United States? A cross-temporal meta-analysis of social dilemmas (1956\u20132017)<\/a>. Psychological Bulletin, 148(3-4), 129.<\/li>\n<\/ul>\n<p>The error bars are larger than expected because Yuan et al plotted the <a href=\"https:\/\/towardsdatascience.com\/how-confidence-and-prediction-intervals-work-4592019576d8\">prediction confidence intervals, not the parameter confidence intervals<\/a>. I don&#8217;t know why they did that, but whatever. There&#8217;s a few replies that make fun of him, and social science in general, <a href=\"https:\/\/twitter.com\/MetaLevelUp\/status\/1554144437051203584\">like this one<\/a>:<\/p>\n<p><a href=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/anti-protzko-tweet.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-10764\" src=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/anti-protzko-tweet.png\" alt=\"\" width=\"541\" height=\"609\" srcset=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/anti-protzko-tweet.png 541w, https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/anti-protzko-tweet-267x300.png 267w\" sizes=\"auto, (max-width: 541px) 100vw, 541px\" \/><\/a><\/p>\n<p>This is basically a complaint that the r\/r\u00b2 value is too small. One can indeed make such a complaint sometimes, but with this kind of meta-analysis with a temporal moderator, the error is on the critics&#8217; side. The reason is that the r and r\u00b2 values are entirely arbitrary when plotting results from a meta-analysis, and that&#8217;s because they are a function of the statistical precision of studies, not the trend itself.<\/p>\n<p>Let me give you an example. Across centuries, human height has starkly increased. A nice meta-analysis study is: <a href=\"https:\/\/elifesciences.org\/articles\/13410\">A century of trends in adult human height (2016)<\/a>:<\/p>\n<p><a href=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/human-height-gains.webp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-10765\" src=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/human-height-gains.webp\" alt=\"\" width=\"1234\" height=\"1333\" srcset=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/human-height-gains.webp 1234w, https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/human-height-gains-278x300.webp 278w, https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/human-height-gains-948x1024.webp 948w, https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/human-height-gains-768x830.webp 768w\" sizes=\"auto, (max-width: 1234px) 100vw, 1234px\" \/><\/a><\/p>\n<p>So it used to be that tall male populations were about 170 cm and now they are more like 180. A gain of 10 cm or so. A standard deviation for male height is about 7 cm, so this is a gain of about 1.4 d. A huge effect size, that can easily be noticed when you walk into old buildings. You often hit the head on the door frames. The studies we have of height are based on large, representative samples. This is totally unlike in most social science, where samples are generally small, thus yielding low precision. Maybe you see where I am going with this. Suppose we imagine two meta-analyses of human height over time. One based on based on small studies and one on large studies. They could look like this:<\/p>\n<p><a href=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/meta_signal_noise.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-10766\" src=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/meta_signal_noise.png\" alt=\"\" width=\"3000\" height=\"1950\" \/><\/a><\/p>\n<p>We see that the left plot has a lot more noise, though the correlation is still decent at .68. On the right side, the plot is near perfect. What are the effect sizes? Simply go back to 7th grade math. Look at the line of fit. Where does it intersect the y axis at year 1900, and where does it end at year 2000. That&#8217;s right, both plots show that the historical trend <em>is exactly the same, 1 cm\/10 years<\/em>. The only difference is that the studies in the left meta-analysis are a lot less precise. In fact, each study is a mere n=5, and on the right side, n=1000. <strong>The r\/r\u00b2 does not tell you what you want to know here. It is not an estimate of the effect size over time.<\/strong><\/p>\n<p>In this case, we are measuring only means, and it turns out that even with n=5 per study, the left plot has a high level of signal in the scatterplot sense (r = .68). But that doesn&#8217;t have to be the case. If we had instead studied something that can only be less precisely estimated, such as cooperation, then the dots on the left side would be a lot more noisy. So let&#8217;s repeat this exercise, but study something harder to precisely quantify, a correlation between height and income. Let&#8217;s pretend that this correlation has been increasing over the years. We might get something like this:<\/p>\n<p><a href=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/meta_signal_noise2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-10768\" src=\"https:\/\/emilkirkegaard.dk\/en\/wp-content\/uploads\/meta_signal_noise2.png\" alt=\"\" width=\"3000\" height=\"1950\" \/><\/a><\/p>\n<p>There sure is a lot of noise in these meta-analyses. In fact, the sample size for studies on the left side is n=50 and they are n=2000 on the right side. Correlations are just that much harder to estimate where even n=2000 studies will not give you a perfect line of fit in a meta-analysis. Looking more closely as we did before, though, we see that the slopes are the same. What these two meta-analyses are telling us about the historical change is exactly the same: the correlation goes up by about 0.01\/10 years, changing from about 0.1 to about 0.2.<\/p>\n<p>TL;DR John Protzko did nothing wrong.<\/p>\n<div style=\"position: static !important;\">\n<p data-pm-slice=\"1 1 []\"><a href=\"https:\/\/rpubs.com\/EmilOWK\/signal_noise_meta\" target=\"_blank\" rel=\"noopener noreferrer nofollow\">Details of statistics here.<\/a><\/p>\n<\/div>\n<div style=\"position: static !important;\"><\/div>\n<div style=\"position: static !important;\"><\/div>\n","protected":false},"excerpt":{"rendered":"<p>OK, it has been written about already by some others, but I also want to talk about this John Protzko pile-on: About 3 weeks ago I thought: &quot;I wonder if behavior on economic games has been getting more selfish over time. But doing that study would be A LOT of work.&quot; Thankfully, someone went through [&hellip;]<\/p>\n","protected":false},"author":17,"featured_media":10762,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1766],"tags":[383,1755,3158],"class_list":["post-10761","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-math-science","tag-fallacy","tag-meta-analysis","tag-signal-and-noise","entry","has-media"],"_links":{"self":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/10761","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/users\/17"}],"replies":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/comments?post=10761"}],"version-history":[{"count":3,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/10761\/revisions"}],"predecessor-version":[{"id":10775,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/10761\/revisions\/10775"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/media\/10762"}],"wp:attachment":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/media?parent=10761"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/categories?post=10761"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/tags?post=10761"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}