I've written before (here and here) about how psychologists encountered problems when trying to get their hands on colleagues' data for reanalysis. Meta-analyses usually contain longish sections about how the authors decided to include the results from some studies but not others in their calculations. Here's an idea for a new inclusion criterion (or exclusion criterion, if you prefer). Meta-analysts could write to study authors asking for their data. If, despite repeated attempts, they cannot obtain that data after, say, a year, the study in question is not included in the analysis, because there is good reason to suspect the reported results are dodgy. (This does not seem like a good strategy if you're dealing with older research.)
Meta-analysts currently do not have an incentive to work like this, because this slows the research process down enormously. But in the long run, it might improve the accuracy of scientific knowledge.
Nothing as Useful as a Bad Theory
4 years ago
2 comments:
That could introduce even more biases into meta-analyses, which I'm already convinced are to all intents and purposes a pseudo-scientific method of confirming the authors' biases. Just by the nature of things, people are going to be far more likely to be able to get data from people who take their side on contentious issues, people they already know etc. than from people they disagree with. It could thus easily be a method of getting rid of any inconvenient data...
Andrew,
as far as meta-analyses in general are concerned, the ones I've seen are quite explicit about their inclusion criteria. It is thus easy for critics to (a) argue that these criteria should be different or (b) point out if studies that were not included should have been according to the criteria.
As for the specific recommendation, I don't agree with your reasoning. If a researcher I disagree with asked me for data because he's conducting a meta-analysis, I'd make sure he gets them asap, so as not to give him a simple reason to attack me. Also, I'd want as much data that's inconvenient for the view I don't hold to be included.
Post a Comment