Blog

This is a follow-up summary of my new interview with Tara McCarthy.

The problems

Science is a broad cluster of methods and practices used to discover patterns in nature while minimizing the influence of random error and human biases. We have come far the last few hundred years, but we still have very far to go in improving the general approach.

Sources of bias include:

1. Hypothesis generation bias (ideas to be tested reflects worldview).
2. Grant bias (refuse to fund ideas you don’t like).
3. Data access bias (refuse antagonists access to data).
4. Publishing related biases:
   1. Submission bias (submit studies that ‘worked’).
   2. Editorial bias (double standards, sexiness, artificial scarcity).
   3. Reviewing bias (double standards).
   4. Publication pressure bias (publish or perish).
5. Hostile response bias (don’t dare to say X because fear reprisals).
6. Media bias (the media prefers some results over others).

Proposed solutions

1. Changing the political distribution among media people:
   1. For countries with public funding for the media (most European), one could have the public decide how these are distributed. This would enable the public to support media in line with their own views, probably resulting in a more even distribution.
2. Changing the political distribution among scientists:
   1. Blinder hiring procedures – standardized tests, research metrics.
   2. Outreach programs.
3. Changing scientific practices:
   1. No editorial decision – prevent desk rejections and biased reviewer selection.
   2. Need to institute non-human anti-spam methods to prevent the absolute lowest quality submissions (only needed for big journals).
   3. Open review – to reduce biased reviews. If you want to say something, say it in your own name. But this might result in less critical reviews. Trade-off. unsure what is best.
   4. Open data – no data access bias. Makes a lot of sense that everybody should have access to publicly funded data.
   5. Pre-results review – to prevent results bias (Registered Reports).
4. Mandating higher-powered studies and higher evidence bars makes it harder to use questionable research practices to publish false positives in line with one’s worldview.

Other things mentioned

• OpenPsych, my open science publisher.
• The dataset I was declined access to.
• The reliability of and bias in peer review: Wikipedia. Richard Smith. Various studies on reliability.
• Bias seems equally strong for left and right-wing people, according to big meta-analysis and recent study. But left-wingers and women are more likely to unfriend people on social media i.e. less open-minded to others’ ideas — contrary to the personality results (self-reported).
• Big review article on bias in science, which has tons of data about surveys etc. Lots of more stuff from Heterodox Academy.