Publication bias: an editorial problem?
Posted on 31st October 2014 by Nathan Collicott
Background
Publication bias is defined as selective publication depending on the results of the study in question. Classically, studies reporting positive results (i.e. studies reporting a difference between the tested intervention and a comparison intervention) are more likely to be published than studies reporting negative results. Publication bias gives a misleading impression of the efficacy and safety of a treatment because we do not have access to all of the data concerning a particular intervention. With missing data, we may overestimate the benefits of an intervention and underestimate its risks, harming patients.
Publication bias has previously been explained in terms of authors and sponsors of studies failing to submit research reporting negative results to medical journals for publication. However, publication bias may also occur once the journal has been submitted for publication: the peer review process and editorial decision making may favour the publication of studies with positive results. A limited number of studies have previously shown little evidence of publication bias in this setting. Most of these studies were conducted prospectively, potentially influencing the editorial process, while others were limited to either high-impact general journals or specialty journals, possibly limiting the generalizability of the results. The current study aims to address these shortcomings.
Study synopsis
This study involved retrospectively reviewing manuscripts reporting the results of clinical drug trials submitted to 8 journals over a two year period. Journals that agreed to participate included the BMJ and 5 BMJ group specialty journals, as well as Diabetologia and the Journal of Hepatology. Studies were classified according to their results (positive or negative) and their funding (non-industry, industry supported or industry sponsored). The authors then looked at whether the manuscript was published, rejected after peer review or rejected outright.
The main finding of the study is that manuscripts reporting positive results were not more likely to be published. Publication was associated with larger, multi-centre studies with more than 100 participants, and with submission to a specialty, rather than a general, journal.
However, 6 of the 8 journals involved in the study were BMJ group, who have an editorial policy of publishing ‘negative’ studies, as long as they are scientifically important and appropriately conducted. Similarly, it may be that journals willing to participate in this study participated because they were confident that their editorial process was not influenced by the direction of study results.
Finally, the current study did not assess the methodological quality of the manuscripts. It may have been that authors feel that negative papers are less likely to be published and therefore only submit negative studies for publication if they are of the highest quality. If this were true, submitted negative manuscripts would be of higher quality than submitted positive manuscripts. In this scenario, publication bias is present if submitted negative papers, though superior methodologically, are not more likely to be published than positive studies. Having said that, in this study, sample size and whether the study was single or multi-centre did not differ between studies reporting positive and negative results. These two measures could be taken as crude, limited measures of study quality, and this suggests that the negative manuscripts in this study were not of superior quality to the manuscripts reporting positive results.
Take home message
This study adds to earlier work showing the absence of evidence of publication bias in the editorial process by studying the topic retrospectively and incorporating both general and specialty journals. This suggests that publication bias occurs mainly before manuscripts are submitted to journals. The authors suggest that in order to further reduce the possibility of publication bias in the editorial process, editors and peer reviewers could initially be blinded to the results and discussion sections of submitted manuscripts. If the manuscript passed this stage, the full article could then be provided for a final decision. Although an interesting concept, this two stage review process has not been implemented by any journals as yet.
Reference
