{"id":6233,"date":"2016-10-03T19:24:00","date_gmt":"2016-10-03T18:24:00","guid":{"rendered":"http:\/\/emilkirkegaard.dk\/en\/?p=6233"},"modified":"2016-10-03T19:24:00","modified_gmt":"2016-10-03T18:24:00","slug":"could-the-openpsych-journals-qualify-as-outlets-for-osfs-preregistration-challenge","status":"publish","type":"post","link":"https:\/\/emilkirkegaard.dk\/en\/2016\/10\/could-the-openpsych-journals-qualify-as-outlets-for-osfs-preregistration-challenge\/","title":{"rendered":"Could the OpenPsych journals qualify as outlets for OSF’s preregistration challenge?"},"content":{"rendered":"

For those that don’t know, OSF — Open Science Framework — runs a pre-registration challenge where one can win 1000$<\/a>. No small amount of cash for doing more research. But Emil, you already do pre-registered studies. Why don’t you submit these for this challenge?<\/em> There is a list of eligible journals<\/a> and I don’t like any of them.<\/p>\n

Now someone writes to me and asks: Have you considered aligning the journals you run with the preregistration <\/a>set-up of the Open Science Framework?<\/em><\/p>\n

No because I didn’t know it was possible to do this. I suppose I thought these journals were chosen in some open open fashion internally by the OSF team. Guess not!<\/p>\n

There are three criteria for being eligible:<\/p>\n

    \n
  1. Becoming a signatory to the Transparency and Openness Promotion (TOP) Guidelines<\/a>, which commits the journal to conducting a review of the guidelines for potential adoption within one year.<\/li>\n
  2. Surfacing transparent research practices by issuing Badges<\/a> in publications.<\/li>\n
  3. Adopting the Registered Reports<\/a> publication format as a submission option for authors.<\/li>\n<\/ol>\n

    Badges and Registered Reports should not be a problem. We already have pre-regged studies with open data and materials, so all that remains is giving out the badges. I’m currently designing the new website for OpenPsych<\/em><\/a>, and so it’s fairly trivial to add the option to give badges.<\/p>\n

    What about the TOP guidelines? They are:<\/p>\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n
    Criterion<\/b><\/td>\nLevel 0<\/b><\/td>\nLevel I<\/b><\/td>\nLevel II<\/b><\/td>\nLevel III<\/b><\/td>\n<\/tr>\n
    Citation Standards<\/td>\nJournal encourages citation of data, code, and materials, or says nothing<\/td>\nJournal describes citation of data in guidelines to authors with clear rules and examples.<\/td>\nArticle provides appropriate citation for data and materials used consistent with journal’s author guidelines<\/td>\nArticle is not published until providing appropriate citation for data and materials following journal’s author guidelines.<\/td>\n<\/tr>\n
    Data Transparency<\/td>\nJournal encourages data sharing, or says nothing<\/td>\nArticle states whether data are available, and, if so, where to access them.<\/td>\nData must be posted to a trusted repository. Exceptions must be identified at article submission.<\/td>\nData must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication.<\/td>\n<\/tr>\n
    Analytic Methods (Code) Transparency<\/td>\nJournal encourages code sharing, or says nothing<\/td>\nArticle states whether code is available, and, if so, where to access it.<\/td>\nCode must be posted to a trusted repository. Exceptions must be identified at article submission.<\/td>\nCode must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication.<\/td>\n<\/tr>\n
    Research Materials Transparency<\/td>\nJournal encourages materials sharing, or says nothing<\/td>\nArticle states whether materials are available, and, if so, where to access them.<\/td>\nMaterials must be posted to a trusted repository. Exceptions must be identified at article submission.<\/td>\nMaterials must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication.<\/td>\n<\/tr>\n
    Design and Analysis Transparency<\/td>\nJournal encourages design and analysis transparency, or says nothing<\/td>\nJournal articulates design transparency standards<\/td>\nJournal requires adherence to design transparency standards for review and publication<\/td>\nJournal requires and enforces adherence to design transparency standards for review and publication<\/td>\n<\/tr>\n
    Study Preregistration<\/td>\nJournal says nothing<\/td>\nJournal encourages preregistration of studies and provides link in article to preregistration if it exists<\/td>\nJournal encourages preregistration of studies and provides link in article and certification of meeting preregistration badge requirements<\/td>\nJournal requires preregistration of studies and provides link and badge in article to meeting requirements.<\/td>\n<\/tr>\n
    Analysis Plan Preregistration<\/td>\nJournal says nothing<\/td>\nJournal encourages preanalysis plans and provides link in article to registered analysis plan if it exists<\/td>\nJournal encourages preanalysis plans and provides link in article and certification of meeting registered analysis plan badge requirements<\/td>\nJournal requires preregistration of studies with analysis plans and provides link and badge in article to meeting requirements.<\/td>\n<\/tr>\n
    Replication<\/td>\nJournal discourages submission of replication studies, or says nothing<\/td>\nJournal encourages submission of replication studies<\/td>\nJournal encourages submission of replication studies and conducts results blind review<\/td>\nJournal uses Registered Reports as a submission option for replication studies with peer review prior to observing the study outcomes.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    A quick count:<\/p>\n

      \n
    1. Citation Standards: 0
      \nThis does not seem necessary.<\/li>\n
    2. Data Transparency: 2\/3
      \nOnly one reviewer conducts independent analytic replications (me), but I hope more will follow. But all share data at OSF repos when possible. Ironically, OSF prevented one paper from posting the data there due to accepting a false copyright complaint.<\/li>\n
    3. Analytic Methods (Code) Transparency: 2\/3
      \nSame as above. We don’t actually have a requirement for sharing code, but it has been done most of the time. The exceptions are because the authors were not familiar with OSF.<\/li>\n
    4. Research Materials Transparency: 2\/3
      \nSame as above.<\/li>\n
    5. Design and Analysis Transparency: ?
      \nUnclear what this means.<\/li>\n
    6. Study Preregistration: 1
      \nBut 2 is possible. 3 is not practically possible as many studies rely on already gathered data.<\/li>\n
    7. Analysis Plan Preregistration: 1
      \nAs above.<\/li>\n
    8. Replication: 1
      \nBut 2 and 3 are possible in the future.<\/li>\n<\/ol>\n

      Scores: 0, 2.5, 2.5, 2.5, ?, 1, 1, 1. A mean of 1.5. Unclear whether this mean OpenPsych is eligible or not.<\/p>\n

      Also, I note that the TOP guidelines overlap with the other two requirements. That seems redundant.<\/p>\n","protected":false},"excerpt":{"rendered":"

      For those that don’t know, OSF — Open Science Framework — runs a pre-registration challenge where one can win 1000$. No small amount of cash for doing more research. But Emil, you already do pre-registered studies. Why don’t you submit these for this challenge? There is a list of eligible journals and I don’t like […]<\/p>\n","protected":false},"author":17,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1107],"tags":[2302,2413,2414],"class_list":["post-6233","post","type-post","status-publish","format-standard","hentry","category-science","tag-openpsych","tag-osf","tag-preregistration","entry"],"_links":{"self":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/6233","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/users\/17"}],"replies":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/comments?post=6233"}],"version-history":[{"count":1,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/6233\/revisions"}],"predecessor-version":[{"id":6234,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/6233\/revisions\/6234"}],"wp:attachment":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/media?parent=6233"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/categories?post=6233"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/tags?post=6233"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}