Consider all the relevant fair comparisons
Posted on 13th December 2017 by Neelam Khan
This is the twentieth blog in a series of 36 blogs based on a list of ‘Key Concepts’ developed by an Informed Health Choices project team. Each blog will explain one Key Concept that we need to understand to be able to assess treatment claims.
We’ve already established the importance of fair comparisons in blog 2.1 but now let’s give some thought to how many comparisons should be taken into account. No prizes for guessing, of course, that every relevant fair comparison should be considered, but why? This blog will explore the importance of considering all the relevant fair comparisons and what can happen if you don’t.
Quantity as well as quality
Would you ever expect a doctor to prescribe medication on the basis of a single successful trial? Rarely, if ever! A single study, regardless of how well-conducted it may be, is unlikely to produce irrefutable evidence for a treatment. Likewise, conclusions cannot be drawn from just a single comparison of treatments as the results can be misleading because of the play of chance.
This is especially true in the case of small studies where few treatment outcomes have been observed. Pooling data together, for instance in the form of a meta-analysis, enables us to carry out statistical tests and determine confidence intervals using the whole body of relevant evidence. Essentially, this helps establish the probability that treatment effects were due to chance and keeps us from wrongly concluding that there are/aren’t statistically significant differences between two treatments.
Taking into account all relevant fair comparisons helps generate more data about the treatments in question. In practice, this means researchers and healthcare professionals can estimate more precisely – and reliably – the effects a treatment is likely to have, be that information relating to the most efficacious dosage, common side effects, or likely success rate.
What are the consequences of cherry-picking comparisons?
Failing to consider all the relevant fair comparisons can have serious consequences. It can mean that participants in research studies and patients are exposed to treatments that may cause them harm, or be denied beneficial treatments.
This is true for the many pregnant women who gave birth to premature babies in the 1980s and ‘90s. Premature babies are often born with lungs and other organs that are not fully developed. One consequence is that neonatal respiratory distress syndrome (NRDS), which is potentially fatal, can result.
A substantial amount of convincing evidence emerged in the 1980s to support the use of corticosteroids in pregnant women expected to give birth prematurely to decrease the incidence of NRDS and the death rate of babies. Despite this, trials continued to be conducted over the subsequent decade in which some participants received a placebo or no treatment – at the cost of their babies’ lives in some cases.
Of course, researchers intended no harm for their participants, they were surely investigating what they believed to be a valid question with the aim of improving the lives of many. It would be unfair to categorise their research as ‘bad’ – perhaps ‘unnecessary’ is more appropriate. If data from all the relevant fair comparisons had been combined earlier (i.e. in a systematic review with meta-analysis), it would have been apparent that further research regarding this treatment would be both unwarranted and unethical. Therefore, considering all the relevant fair comparisons can help to protect patients and research participants from avoidable harm.
Yet more unnecessary research
The above is an extreme illustration of what can happen when all the relevant fair comparisons aren’t considered; but there are also more moderate examples from which we can learn.
Evening primrose oil was touted as a safe and effective treatment of eczema in the early 1980s. There was some sound logic behind the assertion: evening primrose oil contains the essential fatty acid gamma-linolenic acid (GLA), in which people suffering from atopic eczema were believed to be deficient.
Several studies were conducted subsequently, producing contradictory results. Worryingly, much of the published research was sponsored by manufacturers of evening primrose oil supplements. Even a meta-analysis of trials assessing the efficacy of evening primrose oil treatment for atopic eczema in 1989, omitted a large trial which had not found any difference between evening primrose oil and a placebo! In fact, it wasn’t until the mid-1990s that a comparison of all the relevant fair trials was performed by a group of independent researchers, but even this was suppressed by the commissioners.
Nevertheless, fast forward a few years and you’ll find that the Medicines and Healthcare products Regulatory Agency withdrew the marketing license for evening primrose oil as a treatment for eczema. Why? A review of all the relevant fair trials – 20 in total, published and unpublished – finally made it to publication in 2002 and concluded that there is no evidence that evening primrose oil is beneficial for eczema.
Although there was no known harm in using evening primrose oil to treat eczema, as in the previous case of the premature babies with NRDS, valuable resources were wasted. Extensive research was carried out to determine the most efficacious dosage, investigate any possible side effects and explore the potential of similar plant-based products for treating atopic eczema. All the time, effort and money invested in such research could have been directed somewhere more beneficial had all the relevant fair comparisons been considered earlier.
So what?
There are countless instances where not all of the relevant fair comparisons were considered. Reporting all of them would result in an impossibly lengthy blog but, ultimately, the key messages from all of them can be summarised in a few bullet points. Quite simply, it is important to consider all the relevant fair comparisons because it helps to:
- quantify the uncertainty in treatment outcomes,
- provide more reliable and statistically precise estimates of the effects of treatments,
- protect patients and research volunteers from avoidable harm, and
- prevent waste of resources in research and healthcare.
- The Evidence-Based Research Network (EBRNetwork.org) has been established to address these issues
