Tips on reporting clinical trials (not just for journalists)
Posted on 15th July 2013 by Alice Buchan
The reporting of trials in the media isn’t always clear to the reader, and can often be misleading. To promote good reporting of trials; Alltrials have compiled 11 tips for journalists [1], but these tips can also be useful for anyone reading or writing about clinical trials.
So here they are:
- Was this trial registered before it began? It should have been! So that we can scrutinise it and make sure negative results don’t get hidden.
- Is the primary outcome reported in the paper the same as the primary outcome specified in the protocol? If not, why not? Though researchers may publish their protocol in advance when registering a trial, not all registries require it.
- Look for other trials by this company, or group, or on this treatment and on registries: have these all been published? If not, then this report possibly represents a biased and cherry picked finding.
- ALWAYS mention who funded the trial – it matters. Whether some of the ethics committee people have some interest with the funding company.
- Look at whether the country where the work is done will benefit from the trial. Will they get the drug at a lower cost or not? Is it investing a disorder or disease that is a problem in that country.
- How many patients were on the trial, and how many were in each arm of the trial. Some trials can only have small patient groups, but others will need to study many more patients to draw the conclusions that may be being claimed.
- What was being compared? Was it drug vs placebo? Drug vs standard care? Drug with no control arm?
- Be precise about the sort of people or patients who benefited – was it just in advanced disease, people with a particular form of a disease? Contextualise how common these patients are in the bigger picture.
- Report natural frequencies as these are much more meaningful to both lay-people and experts: 13 people per 1000 experienced x is clearer than 1.3% of people experienced x
- Avoid relative risks and try to paint the findings in meaningful terms. So if you say the drug improved survival by X% it’s hard to know quite what that means. If you wrote: people taking the drug lived two months longer on average, you’d be clearer. Best of all would be to say: patients taking the drug lived two months longer on average, which is four months longer than the control group.
- I realise now that I left out the eleventh tip, which is to always seek advice from another expert, who is not involved in the trial and has no vested interest. This is true for reporting any story, but it is worth reiterating here.
The tips were originally tweeted by Ruth Francis, Head of Communications at BioMed Central (@Roobina).
To me, these tips split neatly into 3 groups: ones about the conduct, funding, and possibly sources of bias in the trial, ones regarding PICO, and ones that advise on clarity of reporting – particularly important in the popular press. Tips 1-4 and 11 focus on the conduct of the trial, including funding, and help you pick up any discrepancies (e.g. change in primary outcome) that might warrant considering the results presented very carefully. It is important to identify funding and sources of bias – who might have an interest in the trial’s outcome? This is also where the 11th tip is very important – an expert opinion that is as unbiased as possible can provide an interesting perspective, and point out flaws or strengths that might otherwise be missed. The PICO (Patient/Problem, Intervention, Comparison, Outcome) framework helps ask focused questions in clinical trials, and is equally important when reporting them and considering their effects. The tips regarding PICO and trial design, conduct, and funding are also important when considering a trial as a medical student (or doctor!). The third and final group is probably more relevant to the press than to medics, but when explaining a trial, especially to a non-medic, clarity is key.
Top tips in action
I’m just going to use a recent news article to illustrate the difference these tips can make to clear and balanced reporting of clinical trials; for more critiques of health reporting, take a look at David’s blog series: Health in the media: the good, the bad, and the ugly.
I’m looking at is this recent piece in the Guardian about increased cardiovascular risk in patients taking diclofenac [3] (original study here [4]). The BBC’s piece on the same study has already been critiqued by David here.
Trial design, conduct, funding and bias:
1. Was it registered? As a meta-analysis, this study did not need to be registered before it began.
2. Consistent primary outcome? Not mentioned in the article, but the major outcomes (major vascular events, major coronary events, stroke, heart failure, mortality, and upper GI complications) are the same in the methods and results.
3. Are other trials by this group published? Again, not mentioned, but a quick search of PubMed for the Coxib and traditional NSAID Trialists’ (CNT) Collaboration reveals only one item. I personally wouldn’t be too surprised that this specific collaboration has not published other papers given the scale of this review, and the number of people involved (32 on the writing committee alone, and still more authors who contributed data).
4. Who funded the trial? The Guardian mentioned that the MRC Clinical Trial Service unit ran it, so indirectly they mentioned MRC funding, but this could have been clearer. They also failed to mention British Heart Foundation funding.
11. Opinion from an (impartial) expert? Yes, the Guardian contacted Marie Griffin from Vanderbilt University in the US, who was not connected with the trial. They also had a quote from Prof. Alan Silman of Arthritis Research UK, but he evidently has a vested interest (diclofenac is commonly given to relieve the pain of arthritis).
PICO
5.Where was the trial done? Will people there benefit? As this is a meta-analysis, it’s a bit different, but this UK analysis is likely to benefit patients here, though the trials included are from
6. How many patients? How many in each arm? As mentioned in the Guardian’s article, this meta-analysis pooled results from more than 353,000 patients.
7. What was being compared? Was it drug vs placebo? Drug vs standard care? Drug with no control arm? As detailed in the review itself, this meta-analysis included both NSAID vs placebo and NSAID vs other NSAID trials, though this was not mentioned in the news article. The Guardian don’t make the distinction too clear between the NSAIDs in general and coxibs specifically.
8. What sort of patients will benefit? Specifics! These findings will, as discussed in the Guardian, largely benefit patients with arthritis, but also will benefit patients taking high dose NSAIDs in general.
Presenting data clearly
9. Did they use natural frequencies? As you can see here: “The research found that for every 1,000 people with a moderate risk of heart disease allocated to one year of treatment with high-dose diclofenac (150mg daily) or ibuprofen (2,400mg daily), about three would experience an avoidable heart attack, of which one would be fatal.” , the data are very clearly presented in meaningful natural frequencies, which are easy for people without a statistical background to understand.
10. Did they avoid using relative risks? No, but this is still fairly clear: “The study also found that all Nsaids double the risk of heart failure and produce a two-to-four times increased risk of serious upper gastrointestinal complications, such as bleeding ulcers.”
Overall
My overall impression of the Guardian article discussed above is fairly positive; the reporting is generally clear, fair, and fairly comprehensive. I hope I’ve shown that this checklist is a great tool to use when writing about trials (although I’ve stretched it to a meta-analysis here), and also when reading both trials and reports about them.
Links:
[1] Ruth Francis. The challenges for journalists writing about clinical trials. Available here: http://www.alltrials.net/2013/the-challenges-for-journalists-writing-about-clinical-trials/
[2] Ruth Francis. Ten top tips for reporting on clinical trials.
[3] Press Association. Painkillers can increase risk of heart attack, study says. The Guardian. 30 May 2013. Available here: http://www.guardian.co.uk/science/2013/may/30/painkillers-heart-attack-risk-study
[4] Coxib and traditional NSAID Trialists’ (CNT) Collaboration. Vascular and upper gastrointestinal effects of non-steroidal anti-inflammatory drugs: meta-analyses of individual participant data from randomised trials. Lancet. 2013 (in press). Available online at: http://www.sciencedirect.com/science/article/pii/S0140673613609009
No Comments on Tips on reporting clinical trials (not just for journalists)