Clear Language, Clear Mind

September 11, 2017

Sources of bias in science (comments on Tara McCarthy interview 2017/09/11)

Filed under: Science,Science — Tags: , , , , , — Emil O. W. Kirkegaard @ 22:57

This is a follow-up summary of my new interview with Tara McCarthy.

The problems

Science is a broad cluster of methods and practices used to discover patterns in nature while minimizing the influence of random error and human biases. We have come far the last few hundred years, but we still have very far to go in improving the general approach.

Sources of bias include:

  1. Hypothesis generation bias (ideas to be tested reflects worldview).
  2. Grant bias (refuse to fund ideas you don’t like).
  3. Data access bias (refuse antagonists access to data).
  4. Publishing related biases:
    1. Submission bias (submit studies that ‘worked’).
    2. Editorial bias (double standards, sexiness, artificial scarcity).
    3. Reviewing bias (double standards).
    4. Publication pressure bias (publish or perish).
  5. Hostile response bias (don’t dare to say X because fear reprisals).
  6. Media bias (the media prefers some results over others).

Proposed solutions

  1. Changing the political distribution among media people:
    1. For countries with public funding for the media (most European), one could have the public decide how these are distributed. This would enable the public to support media in line with their own views, probably resulting in a more even distribution.
  2. Changing the political distribution among scientists:
    1. Blinder hiring procedures – standardized tests, research metrics.
    2. Outreach programs.
  3. Changing scientific practices:
    1. No editorial decision – prevent desk rejections and biased reviewer selection.
    2. Need to institute non-human anti-spam methods to prevent the absolute lowest quality submissions (only needed for big journals).
    3. Open review – to reduce biased reviews. If you want to say something, say it in your own name. But this might result in less critical reviews. Trade-off. unsure what is best.
    4. Open data – no data access bias. Makes a lot of sense that everybody should have access to publicly funded data.
    5. Pre-results review – to prevent results bias (Registered Reports).
  4. Mandating higher-powered studies and higher evidence bars makes it harder to use questionable research practices to publish false positives in line with one’s worldview.

Other things mentioned

 

May 21, 2017

Open science: historical perspectives

Filed under: History — Tags: — Emil O. W. Kirkegaard @ 09:53

After the recent London conference on Intelligence, we visited the Science Museum in South Kensington. Unfortunately, we did not have enough time to explore everything, but we did spend some time in the flight section. A number of the texts on display highlight the importance of open science, and are worth posting here.

Stalling science for 50 years because of one hoarder…

Wrong results published, not a new thing.

Patents, holding back science, not a new thing. See also the story of how James Watt held back progress with his 20 year patent on the steam engine (Against Intellectual Monopoly).

Good guy Hargrave.

March 16, 2015

How to integrate Winnower, blog posts and ResearchGate?

Filed under: Metascience — Tags: , , , — Emil O. W. Kirkegaard @ 20:56

Abstract

I argue that traditional scientific publication is extremely costly and that scientific publication must move towards more rapid publication practices. I discuss how this might be accomplished by integrating blogposts, The Winnower and ResearchGate.

Introduction

There are a number of important dimensions that scientists consider when they choose how and where to publish their work. One thing is publicity, aka getting others to read your work. Generally, scientists have relied on high impact journals to ‘make waves’, but they are increasingly using other routes such as ResearchGate, Twitter, blogs and so on. Aside from seeing them in the references, most new papers I read, I find via Twitter, Google Scholar (by looking up a specific researcher or because I get alerts on new publications) and ResearchGate. Very few are found via journal sites. Scientific publishing is changing. It is returning more to the root verb, to make something public, as opposed to publishing as in the publishing industry. The reason is obvious: the internet has made it possible to side-step the middle man entirely. Publishing no longer requires printing stuff on paper. Publishing is near-free. This is the same change that is happening to the recording and movie industries [7].

The other part of sciencing is getting academic credit. Right now this is primarily measured in publications and their citations. However, alternative ways of measuring publication impact are growing fast, named altmetrics.

Legacy publishing is slow and slow is bad

Traditional pre-publication peer review serves a dual purpose: 1) it solves the credit problem: published articles count and so do their citations in journals, and 2) it gives you publicity.

Researchers pay a price for this solution. Publishing is slow, very slow. Typical time spent in peer review for popular journals is often around 9 months (in psychology at least). This is a 9 month delay on science being shared with others. Imagine if emails were 9 months delayed. Engineeringly speaking, this is a crazy inefficiency. We can do better. We need to build a better scientific system for work flow.

There are various proposed solutions. We are currently experimenting with open forum reviewing, which has been noticeably faster so far. However, it may not be just fast enough given that some scientists publish more much than others [1,2]. In fact, in a huge dataset of publications covering 9 years, about 70% of scientists published only a single paper. In another dataset, only 1% of scientists published a paper every year for 16 years in a row. These super-scientists publish 42% of all papers and reap 87% of all citations. Clearly, there are some large differences between scientists. It seems prudent, therefore, that we avoid slowing them done by slow and burdensome peer review [3].

So, assuming that publishing changes over to some kind of post-publication review system á la Winnower, how do scientists get attention in the meantime? One can post links on Twitter, but it would be really neat if papers were automatically exported as PDF from Winnower to ResearchGate too. However, that brings us to…

Papers without paper

The above solution relies upon the PDF format. A PDF is basically an image of a paper with searchable text (sometimes!). It artificially divides text into ‘pages’ even for text where there is no intention to ever print it on A4-sized paper sheets. In general, a PDF compares poorly to HTML. HTML can embed interactive figures which I think will be very big in science in time to come. Probably, the PDF format will get changed so as to accommodate some new things, but it cannot keep up. PDF is an anachronism.

But then, how shall we automatically export output from blogs/Winnower to ResearchGate etc? We need something citeable, maybe with a DOI [4]. We need some kind of unit of scientific publication. A unit that isn’t a PDF. It needs to be a format that can handle moving images, interactive figures and sound. For practical purposes, it should be packed into a single file. A kind of stand-alone copy of a website. One could save the page, however, this does not produce a single file (try and see). One could take a screenshot of the entire post/article/page (a distinction to become more blurred/nonexistent in the future). This however would make it unsearchable. Perhaps some kind of format can be found, otherwise it must be created.

Future scientific work flow?

Given the above, the future work flow of doing and publishing science with Winnower may look something like this:

winnower science flow

Problems to fix

Before we go there, some problems need to be fixed.

Right now, Winnower posts that have not been finalized do not receive a DOI. They don’t even show citation information. To get these, one needs to finalize the post. However, this freezes it, so no future changes can be made. This needs to change if possible, so that it is citeable from the beginning. Otherwise we are introducing a delay again. Science can move faster, if we let it.

Google Scholar indexes Winnower and fast too — very good. However, it does it incorrectly. It ends looking like this [5]:

winnower index

  • The link goes to the PDF file, not the superior HTML version.
  • The description is automatically generated and inserted, and is as a result often nonsensical. The chosen text is not even from the beginning of the post.

Other problems include:

  • No journal information is indexed. This may be a bug. We cannot seem to get this to work for Scholar with Open Psych even though we use Google Scholar’s own metatags.
  • The images are not hosted on Winnower, but link to the blog. This means that if the blog goes down, so do the images. The content is not properly persistent in its HTML form (PDF will be, I guess). Server space wise I see why they may have gone with this solution. However, for persistence, it must be changed sooner or later.
  • The Winnower article does not link back to the blog source.

Some of these problems may be because Scholar indexes the unfinished version.

There are formatting problems when converting from WordPress. If one uses the WordPress ‘pre’ style, and blockquotes, Winnower ignores this and instead produces a wall of text like in this publication [6].

All in all, publishing on Winnower is an interesting gamble for an up-and-coming researcher. It is not known much how credit (older) colleagues will give for content not published in legacy journals. If Winnower fails for whatever reason, one might have ‘wasted’ a lot of good scientific material that could have been used to write old-school papers to send to legacy journals, as well as precious time. For this reason, Winnower will probably attract quite a number of mavericks in the beginning. Every time someone does something like this, fringe ideas spring forth (some of mine are fringe compared to public opinion). This is not necessarily bad, as long as they do not take over. For a project like this to work, it must be shown that it can work just like the regular system does, with regular type science. Only then will the more cautious, risk-aversive or conservative mainstream scientists follow (or die).

References

1. Ruiz-Castillo, J., & Costas, R. (2014). The skewness of scientific productivity. Journal of Informetrics, 8(4), 917-934.

2. Ioannidis, J. P., Boyack, K. W., & Klavans, R. (2014). Estimates of the continuously publishing core in the scientific workforce. PloS one, 9(7), e101698.

3. Björn Brembs, The cost of the rejection-resubmission cycle, The Winnower 2:e142497.72083 (2015). DOI: 10.15200/winn.142497.72083

4. Tal Yarkoni, Now I am become DOI, destroyer of gatekeeping worlds, The Winnower 2:e142557.78481 (2015). DOI: 10.15200/winn.142557.78481

5. Kirkegaard, E. O. W. (2015). S and G in Italian regions: Re-analysis of Lynn’s data and new data.

6. Kirkegaard, E. O. W. (2015). Indian states: G and S factors.

7. Engström, C., & Falkvinge, R. (2012). The Case for Copyright Reform.

January 4, 2015

Publication bias index by the author: a way to keep scientists honest?

Filed under: Metascience — Tags: , — Emil O. W. Kirkegaard @ 03:40

Would this work? Authors can reduce their publication bias measure by publishing new studies that are more honest (with regards to reporting, research practices or not making data up, etc). The primary problem seems, it seems to me, is that authors who currently have a high publication bias index would try to lower it quickly by publishing papers with artificially bad results. Perhaps by attempting to replicate other studies they don’t like and only publishing those that failed to reach the magic p-level.

Critics of null hypothesis testing will of course say that these kinds of problems are inherent to using that kind of statistics (e.g. Harlow et al 1997 or various Bayesian critics e.g. Kruschke 2010). They may be correct, but it seems to me that the focus on publication bias in many recent meta-analysis have helped the problem quite a bit. New methods are being developed to estimate true population effect sizes using only biased results (e.g. van Assen et al, 2014).

It would be best if the index could be automatically calculated by data mining papers by authors. This seems a little beyond my computer science abilities right now. Science basically needs WATSON to do meta-analysis for them, including coding all the data from papers. However, it is surely possible to create a website where information can be filled out for/by (if by, they will surely cheat again!) authors so their indexes can be calculated. I could quite possibly set up this website in 2015.

In general, from my reading of various kinds of meta-science the conclusion is clear: Humans cannot be trusted to do science properly. I don’t even trust myself to do it properly. Presumably, the two best remedies are: 1) making scientists (and humans in general) smarter by genetic methods (by selection or engineering), 2) developing AI to do it for/with (implants) us. Both of these are coming along but the first is met with considerable political opposition. In the meanwhile, the only thing we can do is to try to make the scientific process more robust to human follies, which means that we need to open everything up, i.e. Open Science. This was one of the goals of founding OpenPsych, to push psychology, specifically differential psychology, behavior genetics and the like, in that direction. We seem to have made some progress. Hopefully 2015 will outperform 2014. I am hopefully optimistic.

Refs
van Assen, M. A., van Aert, R., & Wicherts, J. M. (2014). Meta-Analysis Using Effect Size Distributions of Only Statistically Significant Studies.

Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests?. Psychology Press.

Kruschke, J. (2010). Doing Bayesian data analysis: A tutorial introduction with R. Academic Press.

September 29, 2014

Predatory journals

Filed under: Education,Science,Science,Science — Tags: , , , , , , — Emil O. W. Kirkegaard @ 17:48

I had my first Twitter controversy. So:

I pointed out in the reply to this, that they don’t actually charge that much normally. The comparison is here. The prices are around 500-3000 USD, with an average (eyeballed) around 2500 USD.

Now, this is just a factual error, so not so bad. However…

If anyone is wondering why he is so emotional, he gave the answer himself:

A very brief history of journals and science

  • Science starts out involving few individuals.
  • They need a way to communicate ideas.
  • They set up journals to distribute the ideas on paper.
  • Printing costs money, so they cost money to buy.
  • Due to limitations of paper space, there needs to be some selection in what gets printed, which falls on the editor. Fast forward to perhaps 1950’s, now there are too many papers for the editors to handle, and so they delegate the job of deciding what to accept to other academics (reviewers). In the system, academics write papers, they edit them, and review them. All for free.
  • Fast forward to perhaps 1990 and what happens is that big business takes over the running of the journals so academics can focus on science. As it does, the prices rise becus of monetary interests.
  • Academics are reluctant to give up publishing in and buying journals becus their reputation system is built on publishing in said journals. I.e. the system is inherently conservatively biased (Status quo bias). It is perfect for business to make money from.
  • Now along comes the internet which means that publishing does not need to rely on paper. This means that marginal printing cost is very close to 0. Yet the journals keep demanding high prices becus academia is reliant on them becus they are the source of the reputation system.
  • There is a growing movement in academia that this is a bad situation for science, and that publications shud be openly available (open access movement). New OA journals are set up. However, since they are also either for-profit or crypto for-profit, in order to make money they charge outrageous amounts of money (say, anything above 100 USD) to publish some text+figures on a website. Academics still provide nearly all the work for free, yet they have to pay enormous amounts of money to publish, while the publisher provides a mere website (and perhaps some copyediting etc.).

Who thinks that is a good solution? It is clearly a smart business move. For instance, popular OA metajournal Frontiers are owned by Nature Publishing Group. This company thus very neatly both makes money off their legacy journals and the new challenger journals.

The solution is to set up journals run by academics again now that the internet makes this rather easy and cheap. The profit motive is bad for science and just results in even worse journals.

As for my claim, I stand by it. Altho in retrospect, the more correct term is parasitic. Publishers are a middleman exploiting the the fact that academia relies on established journals for reputation.

Review: The Digital Scholar: How Technology is Transforming Academic Practice (Martin Weller)

Filed under: Education,Science — Tags: , , , — Emil O. W. Kirkegaard @ 17:16

https://www.goodreads.com/book/show/12582388-the-digital-scholar

http://gen.lib.rus.ec/book/index.php?md5=5343D586EEEFB7BB24DE5B71FBD07C32

Someone posted a nice collection of books dealing with the on-going revolution in science:

So i decided to read some of them. Ironically, many of them are not available for free (contrary to the general idea of openness in them).

The book is short at 200 pages, with 14 chapters covering most aspects of changing educational system. It is at times long-winded. It shud probably have been 20-50 pages shorter. However, it seems fine as a general introduction to the area. The author shud have used more grafs, figures etc. to make points. There are plenty of good figures for these things (e.g. journal revenue increases).

September 14, 2014

Costs and benefits of publishing in legacy journals vs. new journals

Filed under: Copyright and filesharing,Psychology,Science — Tags: , , , — Emil O. W. Kirkegaard @ 23:34

I recently published a paper in Open Differential Psychology. After it was published, I decided to tell some colleagues about it so that they would not miss it because it is not published in any of the two primary journals in the field: Intell or PAID (Intelligence, Personal and Individual Differences). My email is this:

Dear colleagues,

I wish to inform you about my paper which has just been published in Open Differential Psychology.

Abstract
Many studies have examined the correlations between national IQs and various country-level indexes of well-being. The analyses have been unsystematic and not gathered in one single analysis or dataset. In this paper I gather a large sample of country-level indexes and show that there is a strong general socioeconomic factor (S factor) which is highly correlated (.86-.87) with national cognitive ability using either Lynn and Vanhanen’s dataset or Altinok’s. Furthermore, the method of correlated vectors shows that the correlations between variable loadings on the S factor and cognitive measurements are .99 in both datasets using both cognitive measurements, indicating that it is the S factor that drives the relationship with national cognitive measurements, not the remaining variance.

You can read the full paper at the journal website: http://openpsych.net/ODP/2014/09/the-international-general-socioeconomic-factor-factor-analyzing-international-rankings/

Regards,
Emil

One researcher responded with:

Dear Emil,
Thanks for your paper.
Why not publishing in standard well established well recognized journals listed in Scopus and Web of Science benefiting from review and
increasing your reputation after publishing there?
Go this way!
Best,
NAME

This concerns the decision of choosing where to publish. I discussed this in a blog post back in March before setting up OpenPsych. To be very short, the benefits of publishing in legacy journals is 1) recognition, 2) indexing in proprietary indexes (SCOPUS, WoS, etc.), 3) perhaps better peer review, 4) perhaps fancier appearance of the final paper. The first is very important if one is an up-and-coming researcher (like me) because one will need recognition from university people to get hired.

I nevertheless decided NOT to publish (much) in legacy journals. In fact, the reason I got into publishing studies so late is that I dislike the legacy journals in this field (and most other fields). Why dislike legacy journals? I made an overview here, but to sum it up: 1) Either not open access or extremely pricey, 2) no data sharing, 3) in-transparent peer review system, 4) very slow peer review (~200 days on average in case of Intell and PAID), 5) you’re supporting companies that add little value to science and charge insane amounts of money for it (for Elsevier, see e.g. Wikipedia, TechDirt has a large number of posts concerning that company alone).

As a person who strongly believes in open science (data, code, review, access), there is no way I can defend a decision to publish in Elsevier journals. Their practices are clearly antithetical to science. I also signed The Cost of Knowledge petition not to publish or review for them. Elsevier has a strong economic interest in keeping up their practices and I’m sure they will. The only way to change science for the better is to publish in other journals.

Non-Elsevier journals

Aside from Elsevier journals, one could publish in PLoS or Frontiers journals. They are open access, right? Yes, and that’s a good improvement. They however are also predatory because they charge exorbitant fees to publish: 1600 € (Frontiers), 1350 US$ (PLoS). One might as well publish in Elsevier as open access for which they charge 1800 US$.

So are there any open access journals without publication fees in this field? There is only one as far as I know, the newly established Journal of Intelligence. However, the journal site states that the lack of a publication fee is a temporary state of affairs, so there seems to be no reason to help them get established by publishing in their journal. After realizing this, I began work on starting a new journal. I knew that there was a lot of talent in the blogosphere with a similar mindset to me who could probably be convinced to review for and publish in the new journal.

Indexing

But what about indexing? Web of Science and SCOPUS are both proprietary; not freely available to anyone with an internet connection. But there is a fast-growing alternative: Google Scholar. Scholar is improving rapidly compared to the legacy indexers and is arguably already better since it indexes a host of grey literature sources that the legacy indexers don’t cover. A recent article compared Scholar to WOS. I quote:

Abstract Web of Science (WoS) and Google Scholar (GS) are prominent citation services with distinct indexing mechanisms. Comprehensive knowledge about the growth patterns of these two citation services is lacking. We analyzed the development of citation counts in WoS and GS for two classic articles and 56 articles from diverse research fields, making a distinction between retroactive growth (i.e., the relative difference between citation counts up to mid-2005 measured in mid-2005 and citation counts up to mid-2005 measured in April 2013) and actual growth (i.e., the relative difference between citation counts up to mid-2005 measured in April 2013 and citation counts up to April 2013 measured in April 2013). One of the classic articles was used for a citation-by-citation analysis. Results showed that GS has substantially grown in a retroactive manner (median of 170 % across articles), especially for articles that initially had low citations counts in GS as compared to WoS. Retroactive growth of WoS was small, with a median of 2 % across articles. Actual growth percentages were moderately higher for GS than for WoS (medians of 54 vs. 41 %). The citation-by-citation analysis showed that the percentage of citations being unique in WoS was lower for more recent citations (6.8 % for citations from 1995 and later vs. 41 % for citations from before 1995), whereas the opposite was noted for GS (57 vs. 33 %). It is concluded that, since its inception, GS has shown substantial expansion, and that the majority of recent works indexed in WoS are now also retrievable via GS. A discussion is provided on quantity versus quality of citations, threats for WoS, weaknesses of GS, and implications for literature research and research evaluation.

A second threat for WoS is that in the future, GS may cover all works covered by WoS. We found that for the period 1995–2013, 6.8 % of the citations to Garfield (1955) were unique in WoS, indicating that a very large share of works indexed in WoS is now also retrievable by GS. In line with this observation, based on an analysis of 29 systematic reviews in the medical domain, Gehanno et al. (2013) recently concluded that: ‘‘The coverage of GS for the studies included in the systematic reviews is 100 %. If the authors of the 29 systematic reviews had used only GS, no reference would have been missed’’. GS’s coverage of WoS could in principle become complete in which case WoS could become a subset of GS that could be selected via a GS option ‘‘Select WoS-indexed journals and conferences only’’. 2 Together with its full-text search and its searching of the grey literature, it is possible that GS becomes the primary literature source for meta-analyses and systematic reviews. [source]

In other words, Scholar covers almost all the articles that WoS covers already and is quickly catching up on the older studies too. In a few years Scholar will cover close to 100% of the articles in legacy indexers and they will be nearly obsolete.

Getting noticed

One thing related to the above is getting noticed by other researchers. Since many researchers read legacy journals, simply being published in them is likely sufficient to get some attention (and citations!). It is however not the only way. The internet has changed the situation here completely in that there are new lots of different ways to get noticed: 1) Twitter, 2) ResearchGate, 3) Facebook/Google+, 4) Reddit, 5) Google Scholar will inform you about new any research by anyone one has cited previously, 6) blogs (own or others’) and 7) emails to colleagues (as above).

Peer review

Peer review in OpenPsych is innovative in two ways: 1) it is forum-style instead of email-based which is better suited for communication between more than 2 persons, 2) it is openly visible which works against biased reviewing. Aside from this, it is also much faster, currently averaging 20 days in review.

Reputation and career

There is clearly a drawback here for publishing in OpenPsych journals compared with legacy journals. Any new journal is likely to be viewed as not serious by many researchers. Most people dislike changes including academics (perhaps especially?). Publishing there will not improve chances of getting hired as much as will publishing in primary journals. So one must weigh what is most important: science or career?

March 5, 2014

The quest for the perfect journal to publish in: The case of psychometrics, differential psychology

Filed under: Science — Tags: , — Emil O. W. Kirkegaard @ 16:50

So you’ve just finished your hopefully good paper. Now comes the question. Where to send it to? Actually, normally you would start by looking at journals to send to before writing. Why? Because journals are not very consistent in their requirements in writing style. There is a huge variety in the ways they want you to list references alone. If you’re using a WIZZIWIG word processor (like Word or Loffice), this means that it is cumbersome to rewrite reference systems. In fact it is cumbersome just writing papers.

But, you’re a clever one, you’ve learned about writing in LATEX and so, of course, you wrote your new paper in that. But wait, many journals only take submissions in Word clones. You’ve just restricted yourself further. Writing in LATEX saves you time in the writing process, but it might increase time due to not being able to find a journal that will take .tex files.

But ignoring issues with the chosen word processor, on what criteria should one ideally select journals to publish in? I can think of a number to begin with:

  • Journal impact factor
  • Open access
  • Publication fees
  • Peer review system (or lack thereof)
  • Speed of publication
  • DOI
  • Indexing
  • Number of readers
  • General likeability and respect of the journal
  • Whether they take LATEX

Journal impact factor (IF) is a number based on how often articles in the journal is cited. Naturally, this creates a positive feedback loop where authors try hard to publish in high IF journals which are read more and so cited more, and so increase their IF, and so competition to publish there increases, and so on forever. For journals that have limited space (even artificial), this also increases their selectivity of papers, i.e. making it harder to publish there. ‘Top’ journals like Nature are super selective because of this.

Open access is obvious. Science should be freely available to all. That’s the way it can best be utilized in practice, e.g. in politics which is notoriously unscientific. If you want more people to read your article and not just the abstract, then open access is a must.

Publication fees are fees that authors have to pay to publish in the journal. Naturally you don’t have to pay those. Sometimes there is an interaction with open access, in that a journal that is normally closed access, will agree to publish your paper open access if you pay a lot of money. “a lot” is not an exaggeration. Intelligence (IF = 2.8), for instance, allows open access but only if you pay 1800 USD plus taxes. Yes, they are absolutely immoral. No wonder, it is an Elsevier journal, a company whose main purpose is leeching money from the scientific community and the public who sponsors the scientific community. Elsevier is very evil.

Peer review is the the practice of having peers i.e. other researchers in the same broad area review your articles. The usual practice (pre-print peer review) is having an editor who receives manuscripts that people send in. He then decides if its utter shit or irrelevant or uninteresting (or something), and if it is he rejects it. If it isn’t (in his opinion), he sends it to some reviewers. These reviewers then maybe after some time write back to the editor of whether they think the article should be published. The editor then decides on those grounds in some kind of idiomatic way whether to publish or not.

In many ways, this is not a good way of doing science. See here and Nosek, Brian A., and Yoav Bar-Anan. “Scientific utopia: I. Opening scientific communication.” Psychological Inquiry 23.3 (2012): 217-243. What you want is instant publication, post-publication open peer review.

Speed of publication is how fast the decision to publish the paper or not is made after the manuscript is sent in. This can take literally years. A typical tactic is after a journal rejects a paper, just to send it to another journal and wait again. After some years of doing this someone will publish the paper unless it is unbelievably bad, and then that might not be enough.

DOI (digital object identifier) is a clever way of quickly referring to any piece of science published, figure, database, article, book, whatever. Many journals provide a DOI for the paper, but some don’t. You want this.

Indexing is if and where the journal’s contents are indexed, e.g. in Google Scholar. Since finding relevant papers via search engines is a very common way of finding papers, you want this.

Number of readers is obvious. You want to be read. Publishing in Icelandic in an unknown journal is not a good idea for this purpose.

General likability and respect of the journal. If you don’t want to be disliked, publishing in allegedly or truly racist or pseudo-scientific journals might not be a good idea.

LATEX, already covered. You want this to save time.

Where to publish?

Given the above criteria, what options are there? I can thin of some:

Journal/Criteria Impact Factor Open Access
Publication Fee Peer review Speed DOI Indexing Readers Likability LATEX
Intel., PAID, etc. High No (with fee only) No Pre Slow to very slow Yes Yes Yes Yes Some of them
JOI Low Yes No Pre Medium Yes ? ? ? Yes
Unknown three journals Very low Yes No Pre ? ? ? No ? ?
MQ Lowish No No Pre Medium No Sporadic Lowish No No

 

I published my first paper in MQ. I don’t mind the general dislike of it, as the works therein are quite serious in my opinion. Pre-print peer review was thorough too. Their lack of LATEX is annoying, but most annoying is the lack of indexing and DOI.

JOI might become good in the future. Right now it is weird. It has a weird system where they apparently only have special issues, and so one can’t submit stuff not covered in the special issue. Very odd.

One can also just self-publish. Perhaps set up a journal oneself to get DOI and stuff.

February 5, 2014

Some ideas about open and decentralized science

Filed under: Science — Tags: — Emil O. W. Kirkegaard @ 14:52

Science is broken

There are numerous errors with science as it is right now. Some of them are: closed access, publication bias, lack of data sharing, the horribly bad and slow publication system.

 

All these problems have done talked about before other places (e.g. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2051047). Here I wanna talk about solutions.

 

Closed access

This one is the most easy one to solve. Make papers (and books) openly available. Thousands of journals already do this, but it is a slow process. This is because the current journals make a lot of money by having their middle man position, so they of course refuse to change. Currently, books are not made openly available, at least legally. Huge pirate sites make millions of books available however. This trend will continue towards all books being openly available, legally or not. A copyright reform will hasten this change.

 

Publication bias

Positive results are usually more interesting and so both journals and authors tend to focus on writing/publishing them. This leads to a systematic bias in the literature which misleads everyone. The solution to this is to create structural incentives to publish exact (not conceptual) replications of other studies. Whatever barriers there are to this need to be removed.

 

One idea is to make a certain type of paper that is termed a replication. It needs to cite the paper it replicates, and a database about replications must be made, so that one can actually find the replications. Right now there is no easy way to sort thru all the papers that cite a particular paper to find those that actually tried to replicate it. There needs to be a category for this.

 

To create the incentive to replicate prior findings, publication indexes (e.g. H-index) need to reward authors of replications.

 

Data sharing

Hundreds of thousands of papers now exist which report statistical results of data. However, the data itself and the exact methods are rarely shared. This means that other researchers cannot rerun the data, or pool the data properly into a larger study (a true meta-analysis). Instead they have to rely on pooling pools of data (standard meta-analysis). There is no way to control for statistical errors authors might have made, or to detect possible data fraud.

 

Think about it this way. When school children hand it mathematics assignments, teachers require them to hand it all the calculations and often the data as well. But scientists don’t do this even tho this is grown ups and the stakes are infinitely higher. It makes no sense at all.

 

By default all data should be shared. The only exceptions are about confidential information, e.g. specific information about correct test items in a test (say, SAT), or names. Names can be anonymized easily.

 

Any science publisher should have a data bank where one can send in data. Given the costs of this nowadays, it is beyond stupid that is isn’t done.

 

Peer review

See the infograph here (http://emilkirkegaard.dk/en/?p=4095). There are various ways to move beyond normal PrePPR (pre-publication peer review) into PostPPR. The major problem is the enormous time that is wasted between first submission and the time when it is available. First it must get thru PrePPR and a number of revisions, and then it must wait for the next issue before other people can see it. Sometimes a paper needs to try a few different journals before it finally gets thru. This is crazy. It should be available to begin with, see the infograph above for another system.

 

The actual peer review is more tricky. There are various ways to do this. One might have a number of editors, who decide collectively which category (rejected, revision needed, accepted) papers belong in. This is close to the current system, and is what is depicted in the #2 system in the above link.

 

However, a true decentralized system is possible. Instead of having a small number of editors, one can make the class much larger. The most decentralized system is just having everybody who wants to rate on papers, and then aggregate those ratings in various ways. The raters themselves can publish have published papers, and one can thus weigh their ratings by that factor. Such a system would be entirely self-sufficient once it got off the ground (critical mass).

 

Besides the rating and categorization of papers above, the publication system merely needs to assign each submitted paper a DOI and a way to cite it:

 

Kirkegaard, Emil OW (2014). ”Radically open and decentralized science”. OPDESC, published 2014-02-05. doi:12.3456/opdesc.2014.02.0001

 

And register it in central systems (e.g. Google Scholar).

November 28, 2011

Reeding material: Peer review, open acces, open sience

Filed under: Copyright and filesharing,Science — Tags: , , — Emil O. W. Kirkegaard @ 12:20

For som reeson i started reeding about peer review, and i ended up reeding a lot of interesting articles.

http://en.wikipedia.org/wiki/Peer_review

Two quotes stand out, copyd from the Criticism section:

“There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.”

“The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability—not the validity—of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong. “

http://www.mja.com.au/public/issues/172_04_210200/horton/horton.html

Alternativs to standard practice and related topics:

http://en.wikipedia.org/wiki/Open_peer_review

http://en.wikipedia.org/wiki/Open_research

http://en.wikipedia.org/wiki/The_Open_Source_Science_Project

http://en.wikipedia.org/wiki/Open_access_%28publishing%29

http://en.wikipedia.org/wiki/Serials_crisis

The obvius paralel:

http://en.wikipedia.org/wiki/Free_and_open_source_software

And, did u no that “gratis” is used in English? I certainly didnt, but it is nice to see that English ‘has lerned’ somthing from the other Germanic languajes :p

http://en.wikipedia.org/wiki/Gratis_versus_libre

Powered by WordPress