Explanations about how treatments work can be wrong
Posted on 18th August 2017 by Cole Wayant
This is the eleventh blog in a series of 36 blogs based on a list of ‘Key Concepts’ developed by an Informed Health Choices project team. Each blog will explain one Key Concept that we need to understand to be able to assess treatment claims.
Putting treatment explanations to the test – how do we know when to trust them?
Maybe you have heard that fish oil supplements can boost schoolchildren’s concentration (1). Maybe, as a woman, you have been offered hormone replacement therapy after menopause to prevent heart attack and stroke (2).
What do the two examples have in common? They both failed to benefit patients as advertised. There were reasons to believe that both treatments should work in theory but, when put to the test, it turned out neither worked in practice.
Fish oil supplements appear to be no more useful than placebo (3, 4) and hormone replacement therapy in post-menopausal women was shown to increase the risk of risk of heart attack and stroke, rather than decrease it (5).
To more accurately evaluate the benefits and harms of treatments, we must understand two things before we make judgements about new medications.
We must pay attention to the data, not the headline
Newspaper headlines are meant to be provocative, not necessarily informative. Ben Goldacre, author of Bad Science, wrote about the importance of looking at data rather than headlines (1). He wrote about a headline in The Observer touting fish oil supplements and their claimed ability to help schoolchildren concentrate. The frenzied media promotion of fish oil supplementation over the years likely contributed to a doubling of global fish oil sales between 2002 and 2007, and a continued boost of eight million new users in the United States between 2007 and 2012 (6).
The benefits of fish oils have been extensively researched and shown to have nowhere near the effect advertised. In fact, the original headline was constructed based on a misinterpretation of the original study. Thus, we are left to believe that this “wonder drug” was a sham. In short, the newspaper article which advocated for the benefits of fish oils was based on a brain imaging study.
Children in this study were given either a big dose of Omega-3, a small dose of Omega-3 or no Omega-3 and they then completed some attention tasks while in a brain scanner to assess whether parts of their brain lit up differently. But what does this really tell us? Goldacre argues that sales reps and researchers often enthusiastically approach doctors with exciting theoretical reasons why a treatment might work. But showing that different parts of the brain light up in a brain scanner is very different from showing that something like Omega-3 has an impact on important, meaningful outcomes in the real world. Indeed well-conducted trials have failed to demonstrate impact of fish oils on health outcomes or long-term cognitive performance (see 1).
We must know when to rely on surrogate endpoints
What is a surrogate endpoint? A surrogate endpoint is an outcome measured in a clinical trial that indirectly measures a patient-relevant outcome (7). For example, in patients at risk for a heart attack, cholesterol levels may be measured as a surrogate endpoint for the risk of heart attack. Sometimes surrogate endpoints are useful, while other times they can be misleading.
It is known that after suffering a heart attack patients may develop an irregular heartbeat (arrhythmia), and that these patients are at higher risk for sudden cardiac death. To prevent sudden cardiac death in heart attack patients, doctors in the 1970s prescribed class I anti-arrhythmics to patients having heart attacks in an attempt to increase survival in heart attack patients. This was a fatal mistake and a clear example of a treatment which should work in theory failing to work in practice. Worse still, in this case the treatment was harmful.
Later studies showed that, compared to placebos, class I anti-arrhythmic drugs doubled the risk of sudden cardiac death in heart attack patients. So while it may have seemed completely reasonable to prescribe these drugs, we relied on a surrogate outcome (presence of arrhythmia) to predict a patient-relevant outcome (sudden cardiac death).
Surrogate endpoints can also be helpful, however. For example, in patients with type I diabetes who measure blood sugar levels to determine the proper insulin dose. The difficult part is knowing when to use surrogate measures and when to be cautious.
To this day there is concern surrounding surrogate endpoints in many fields. In oncology, where new drugs are approved almost monthly, there is an ongoing discussion about the usefulness of surrogate endpoints, since they have been shown to correlate poorly with patient-relevant outcomes (8).
To conclude…
A newly published study can be ‘weaponized’ if the reader is not careful. Misrepresentation of data by the primary authors, misinterpretation by the readers, or misapplication by physicians and guideline developers may negatively affect patient outcomes.
Be aware that claims about treatments may be based only on indirect evidence. For example, a drug might have been shown to have an impact on some surrogate outcome. Although this might suggest the treatment could have a beneficial impact on real-world important outcomes, it does not demonstrate it.
It’s important to remember that treatments that should work in theory may not work in practice and can even turn out to be harmful. A theoretical explanation about how or why a treatment may work is not – in and of itself – proof that it works or is safe. Ultimately we need to put theoretical claims about how treatments work to the test. So we shouldn’t assume that claims about treatments which are based on explanations of how that treatment might work are correct unless there is good quality evidence to back up those claims.
Learning resources which further explain why explanations about how treatments work can be wrong
Read the rest of the blogs in the series here
References (pdf)
Take home message:
