Do treatment comparisons reflect your circumstances?
Posted on 24th May 2018 by Ian Matchett
This is the thirty-fourth blog in a series of 36 blogs based on a list of ‘Key Concepts’ developed by an Informed Health Choices project team. Each blog will explain one Key Concept that we need to understand to be able to assess treatment claims.
Making informed health choices requires considering evidence found in clinical trials, most often in the form of a systematic review. However, not just any systematic review will be relevant – many factors must be considered when assessing whether a review’s findings will apply to any given patient. These considerations range from the relatively straightforward (“does the patient have the same disease process as the patients in the review” or “is the trial treatment available to patients in my area”) to more complex questions (“are the findings of this trial at risk of being biased” or “do the outcomes measured in the review actually represent important changes that the patient will appreciate”).
A very important consideration that often goes unconsidered is whether the clinical trial is examining a treatment under regular circumstances or under conditions which are near perfect – so perfect that regular patients will never experience such optimal conditions. Trials under idealised conditions are more likely to exaggerate the treatment effects when considering how the treatment would fare in the real world, and so may not be a fair representation of how the treatment would affect average patients.
This idea can be further examined by explaining the difference between explanatory and pragmatic trials.
What are explanatory and pragmatic trials?
Randomised Control Trials (RCTs) are commonly accepted to be one of the highest levels of evidence, however consideration must be given to making them cost-effective and time-effective (1). In order to make them worthwhile, they must address a clear question. These questions can be broadly categorised as either explanatory (“does this treatment work under ideal circumstances?”) or pragmatic (“Does this intervention work in the real world with usual standards of care?”) (2).
The ‘ideal circumstances’ of explanatory trials generally consist of carefully chosen patients with a clearly defined disease, doctors highly experienced with the treatment, and resources for intensive monitoring and follow-up (3). These trials aim to maximise the chances of detecting any possible treatment effect, often to demonstrate that the treatment is worthy of further funding or development (4).
On the other hand, pragmatic trials aim to emulate ‘real-world conditions’, including a very broad range of patients to better represent the general characteristics of the population with the disease (1). Treatments are not delivered with any more funding than is available in usual care settings, and no special effort is made to ensure compliance, monitoring, and follow-up (5). Pragmatic trials aim to demonstrate the effect of an intervention delivered in a real-world setting, and as such are considered to be more applicable to usual/average patients.
In reality, pragmatic and explanatory trials are the extremes of a spectrum along which most RCTs sit – an RCT may have very selective inclusion criteria (characteristic of explanatory trials), but the trial may not have the resources to intensively monitor patients and ensure adequate follow-up (characteristic of pragmatic trials). An excellent summary can be found in David Sackett’s 2011 paper ‘Explanatory and pragmatic clinical trials: A primer and application to a recent asthma trial’, reproduced below (5).
Table 1 – Trial elements, illustrating the extremes of the explanatory-pragmatic continuum. Reproduced from Sackett DL. Explanatory and pragmatic clinical trials: a primer and application to a recent asthma trial. Pol Arch Med Wewn. 2011 Aug;121(7–8):259–63.
It is important to consider this spectrum when deciding if a trial is relevant to making an informed health choice because if a trial is pragmatic, it is more likely to be relevant to individual patients within its trial population and reflect the circumstances under which they receive treatment. This means that the findings from the trial are more generalisable to the patient’s situation, whereas an explanatory trial assesses whether an intervention may have any impact at all, but not necessarily upon any given patient. Furthermore, explanatory trials are designed to maximise treatment effects, and so findings from these trials may be exaggerated when compared to administration of the treatment in real-world conditions.
How to tell if a trial is pragmatic or explanatory?
To determine where a trial sits on the pragmatic-explanatory continuum, multiple components of the trial’s design must be taken into account (see table 1). Tools have been developed to help the reader assess all these components and to come up with an overall assessment of the trial – Gartlehner et al’s 2006 paper is an early example (6). However as each component of a trial can have pragmatic or explanatory characteristics, it is important that a tool doesn’t only give an overall rating of the trial (6). The PRECIS-2 tool gives a pictorial depiction of where a trial sits on the continuum in nine individual domains, thus giving the reader a more nuanced view of the trial’s place upon the pragmatic-explanatory continuum (7).
Figure 1 – The PRagmatic-Explanatory Continuum Indicator Summary 2 (PRECIS-2) wheel. Reproduced from Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015 May 8;350:h2147
While the PRECIS-2 tool was designed for use during the trial-design process, it is still useful for the reader as it gives an easy-to-understand, visual depiction of how the trial fares. It assesses nine domains (7):
- eligibility criteria – how similar are the included patients to the general population of patients with this disease?
- recruitment – how much effort beyond usual care was used to recruit patients?
- setting – how different are the trial settings to usual care settings?
- organisation – how different were the resources, expertise, and organisation of the treatment compared to usual care?
- flexibility (delivery) – how different was the flexibility of the treatment delivery compared with usual care?
- flexibility (adherence) – how different was the flexibility of treatment adherence and monitoring to usual care?
- follow-up – how different is the intensity of follow-up compared to usual care?
- primary outcome – how relevant is the trial’s primary outcome to participants?
- primary analysis – how much of the data analysed is directly relevant to the primary outcome?
Each domain is scored from 1 (very explanatory) to 5 (very pragmatic), and is depicted via a radar chart (also known as a spider or star chart) (7). The larger the plotted area, the more pragmatic the trial is. To illustrate this, we will consider two trials.
Trial 1: The randomised controlled trial of self-supervised and directly observed treatment of tuberculosis (DOT) considers whether directly observed swallowing of medication increases compliance amongst South African adults with new-onset pulmonary tuberculosis, compared with usual care (self-administration) (8).
Trial 2: The North American Symptomatic Carotid Endarterectomy Trial (NASCET) considers whether major stroke or death in the next two years is decreased in patients with symptomatic stenosis of a carotid artery if a carotid endarterectomy (performed by an expert vascular or neurosurgeon) is performed along with best medical therapy, compared with best medical therapy alone (9).
Table 2 – Analysis of the DOT and NASCET trials using the domains of the PRECIS-2 tool. Adapted from (7,10).
This visual depiction of the pragmatic-explanatory continuum makes it immediately obvious that one trial (DOT) is more pragmatic than the other (NASCET). While the PRECIS-2 chart will not be formally applied to every trial the reader comes across, it is a useful tool to base your understanding of how applicable the trial is to any individual patient. As such, it will aid in the formation of informed health choices as the patient and/or doctor will have a better understanding that not all RCTs in a given area are equally applicable to any individual patient.
Acknowledgments – I would like to thank Sir Iain Chalmers and Professor Andrew Oxman for their generous help.
