For those that don’t know, OSF — Open Science Framework — runs a pre-registration challenge where one can win 1000$. No small amount of cash for doing more research. But Emil, you already do pre-registered studies. Why don’t you submit these for this challenge? There is a list of eligible journals and I don’t like any of them.
Now someone writes to me and asks: Have you considered aligning the journals you run with the preregistration set-up of the Open Science Framework?
No because I didn’t know it was possible to do this. I suppose I thought these journals were chosen in some open open fashion internally by the OSF team. Guess not!
There are three criteria for being eligible:
- Becoming a signatory to the Transparency and Openness Promotion (TOP) Guidelines, which commits the journal to conducting a review of the guidelines for potential adoption within one year.
- Surfacing transparent research practices by issuing Badges in publications.
- Adopting the Registered Reports publication format as a submission option for authors.
Badges and Registered Reports should not be a problem. We already have pre-regged studies with open data and materials, so all that remains is giving out the badges. I’m currently designing the new website for OpenPsych, and so it’s fairly trivial to add the option to give badges.
What about the TOP guidelines? They are:
| Criterion | Level 0 | Level I | Level II | Level III |
| Citation Standards | Journal encourages citation of data, code, and materials, or says nothing | Journal describes citation of data in guidelines to authors with clear rules and examples. | Article provides appropriate citation for data and materials used consistent with journal’s author guidelines | Article is not published until providing appropriate citation for data and materials following journal’s author guidelines. |
| Data Transparency | Journal encourages data sharing, or says nothing | Article states whether data are available, and, if so, where to access them. | Data must be posted to a trusted repository. Exceptions must be identified at article submission. | Data must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication. |
| Analytic Methods (Code) Transparency | Journal encourages code sharing, or says nothing | Article states whether code is available, and, if so, where to access it. | Code must be posted to a trusted repository. Exceptions must be identified at article submission. | Code must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication. |
| Research Materials Transparency | Journal encourages materials sharing, or says nothing | Article states whether materials are available, and, if so, where to access them. | Materials must be posted to a trusted repository. Exceptions must be identified at article submission. | Materials must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication. |
| Design and Analysis Transparency | Journal encourages design and analysis transparency, or says nothing | Journal articulates design transparency standards | Journal requires adherence to design transparency standards for review and publication | Journal requires and enforces adherence to design transparency standards for review and publication |
| Study Preregistration | Journal says nothing | Journal encourages preregistration of studies and provides link in article to preregistration if it exists | Journal encourages preregistration of studies and provides link in article and certification of meeting preregistration badge requirements | Journal requires preregistration of studies and provides link and badge in article to meeting requirements. |
| Analysis Plan Preregistration | Journal says nothing | Journal encourages preanalysis plans and provides link in article to registered analysis plan if it exists | Journal encourages preanalysis plans and provides link in article and certification of meeting registered analysis plan badge requirements | Journal requires preregistration of studies with analysis plans and provides link and badge in article to meeting requirements. |
| Replication | Journal discourages submission of replication studies, or says nothing | Journal encourages submission of replication studies | Journal encourages submission of replication studies and conducts results blind review | Journal uses Registered Reports as a submission option for replication studies with peer review prior to observing the study outcomes. |
A quick count:
- Citation Standards: 0
This does not seem necessary. - Data Transparency: 2/3
Only one reviewer conducts independent analytic replications (me), but I hope more will follow. But all share data at OSF repos when possible. Ironically, OSF prevented one paper from posting the data there due to accepting a false copyright complaint. - Analytic Methods (Code) Transparency: 2/3
Same as above. We don’t actually have a requirement for sharing code, but it has been done most of the time. The exceptions are because the authors were not familiar with OSF. - Research Materials Transparency: 2/3
Same as above. - Design and Analysis Transparency: ?
Unclear what this means. - Study Preregistration: 1
But 2 is possible. 3 is not practically possible as many studies rely on already gathered data. - Analysis Plan Preregistration: 1
As above. - Replication: 1
But 2 and 3 are possible in the future.
Scores: 0, 2.5, 2.5, 2.5, ?, 1, 1, 1. A mean of 1.5. Unclear whether this mean OpenPsych is eligible or not.
Also, I note that the TOP guidelines overlap with the other two requirements. That seems redundant.