The Older Paper: Data Sharing in Psychology, plus: The Cynical Theory of Academic Journals

Here’s some research that should have made the news: In a note published in 2006, Jelte M. Wicherts, Denny Borsboom, Judith Kats, and Dylan Molenaar report how successful they were when they tried to obtain data from researchers for reanalysis – data which had been the basis of articles that had appeared in journals published by the American Psychological Association (APA). Despite half a year of trying, they obtained a mere 27% of the datasets they had been asking for.

Twenty-seven percent! If you think about it, that’s a scandal. One wonders how trustworthy the results reported in the remaining 73% of papers are.

As proposed by the authors, there is a simple solution to this problem: Journals should require authors who wish to publish with them to hand over the data used in papers generally accepted for publication so that it can be published as an online appendix to the article. (Special arrangements could be made if there are good reasons to not make the data publicly available.)

As this doesn’t seem to be a new problem (see the references in the linked note), one wonders why the vast majority of journals don’t do exactly that anyway. As the solution seems so obvious – I don’t believe editors or publishing houses just didn’t think of this – my interpretation is a cynical one. The authors of articles have an interest in producing an article that is as interesting (i.e., publishable, citable) as possible; this may require some decisions in the data analysis process that do not stand up to scrutiny. Hence authors have no interest in putting colleagues in a position to reanalyze their data and take the results apart. The journal editors face pretty much the same incentives: Their interest is not in pursuing The Truth but in publishing articles that are as citable as possible, as citation counts are what the journal’s reputation hinges on.

One can thus understand why journal editors and publishing houses will not require publication of data. In the case of psychology in particular, however, the situation may be different. If I’m not mistaken, all the most renowned psychology journals are published by APA. APA could thus centrally decide that all journals published by them must adopt the policy described above. On the one hand, APA journals would lose some interesting-looking papers by creative authors, on the other, papers published in APA journals would come to be seen as particularly trustworthy. (A case of what Piotr Sztompka – incorrectly, I think – calls “the paradox of trust”: By institutionalizing mistrust in the papers’ results, readers’ trust in the results would be enhanced.) Although the same argument could be made for individual journals, APA’s market domination could help to make ‘public data’ articles the standard; ideally, an article that is not published in a ‘public data’ journal would come to be regarded like an article that is published in a journal which has no peer review process. Once established in psychology, this practice might spread to other fields.

I’m somewhat surprised this post ends on such an optimistic note.