Are you very different from the people studied?
Posted on 11th May 2018 by Bethan Copsey
This is the thirty-second blog in a series of 36 blogs based on a list of ‘Key Concepts’ developed by an Informed Health Choices project team. Each blog will explain one Key Concept that we need to understand to be able to assess treatment claims.
When you’re examining the results of a systematic review, it is important to look at the individuals it relates to. This is because treatments may have different effects in different individuals.
You need to look at the eligibility criteria of the review – perhaps it only looked at animal studies. Medicines can have different effects in animals and humans – just like chocolate! If you looked at a systematic review on the effect of chocolate on dogs, you would find that it is poisonous and should not be fed to them. This should clearly not be applied to humans!
Similarly, a treatment may work well in animals but be harmful in humans; it is likely that animal studies are conducted earlier in the development of treatments so there could be side-effects in humans which have not yet been recognised.
So it may not always be appropriate to apply the findings from a review to your situation. The eligibility criteria (also called the inclusion / exclusion criteria) tell you how the reviewers decided which studies to include when they reviewed the evidence. This can help you see the similarities and differences between the area you’re interested in and the studies in the review.
This applies not only to differences between humans and animals – treatment effects may differ because of other differing characteristics. For example, a review may search for any study of drugs for the treatment of acne but it may only find studies on mild acne. The treatments which are found to be effective in mild acne may not work for severe acne.
It may also be that the review only considers studies in children when you are more interested in adults. So it is important to take into account how similar your situation is to the people in the studies considered in the review.
You also need to check the eligibility criteria of the included studies. Even if the eligibility criteria of the review applies to you, the included studies may be more selective. For instance, a review of the effects of a drug on a person’s breathing may include any human study of the drug, but if all of the included studies only focus on elite athletes, you should still be cautious about applying the findings to people in general.
Photo by Matt Lee on Unsplash
Similarly, a review looking at the effects of a drug on reducing blood pressure might only find studies involving participants who have both high blood pressure and diabetes. If you don’t have diabetes, a drug may affect your blood pressure differently.
This is part of the ‘Indirectness’ criteria of the GRADE approach where the quality of the evidence provided by a systematic review is downgraded if the populations in the included studies are very selective.
So even if the eligibility criteria for the review fit your situation, you also need to check that the included studies have evaluated participants similar to you. If they are highly selective, the results may not apply to you and could be misleading.
Some population characteristics that may be important to check are:
- Disease severity: A treatment may be very effective in early-stage disease but not in very severely affected
- Age: A treatment may be very effective in adults but harmful in children
- Gender: Treatments may work differently in men and women, or may be unsuitable for pregnant women
- Co-morbidities: A treatment may be less effective or harmful in participants with another condition. Similarly, a drug may produce negative side-effects if it is used with another drug. For example, antibiotics can reduce the effectiveness of oral contraceptive pills.
So when you are using a systematic review to inform your decisions, you need to:
- Check whether the eligibility criteria for studies included in the review apply to your situation, and
- Check if the eligibility criteria of the included studies are very selective
Useful link:
For more information on assessing indirectness, check out these eight introductory slidecasts about using the GRADE approach.
