Tricks to help you get the result you want from your study!
Posted on 9th December 2016 by Sam Marks
I thought about giving this blog a more sensible title. Something like “things to look out for when reading a paper on a clinical trial”. But, I needed to get your attention, didn’t I….
Let’s set the scene: An increasing number of clinical trials are funded by the pharmaceutical industry.
But this is clearly problematic. In 2012, a Cochrane Review including 49 studies, found that industry-sponsored studies testing drugs and medical devices more often favour the sponsor’s products than non-industry sponsored studies.
Mmm, sounds fishy. What might be going on here?
Well, imagine you work in the pharmaceutical industry and you have a new drug. You’ve got a lot of money riding on it, so you need to show that it’s safe and effective. You need evidence to back up your claims that this is the new best thing, and that doctors should prescribe it to their patients. The thing is, your new drug isn’t spectacular. But you really do have a lot of money riding on it.
So what can you do to make sure you get the result you want from your study?
There are many tricks you could try… (and beware, these can be fairly subtle, so even a fairly eagle-eyed reader may not notice them).
Fiddle with the study design…
- You could study the drug in a group of participants you know are likely to respond well to it. For instance, younger people are likely to respond better to medication than older people who are already on lots of medication. So test the drug in those young participants (even if they’re unlikely to be the ones who are actually prescribed the drug in real life).
- Compare your drug against another drug that’s useless. For instance, you could use an inadequate dose of the comparator drug, so that the patients receiving that drug don’t do well. Or, you could give too high a dose of the comparator drug, so that those patients experience considerable side-effects. By comparison, your drug will seem more effective and/or better tolerated.
- Timing is everything. If the difference between your drug and the comparator drug becomes significant after 2 months, stop the trial! You’ve got the results you want and you don’t want to risk the difference disappearing if you let the trial run on. On the other hand, if at 2 months your results are nearly significant, extend the trial by a couple of months, and wait until your results become significant!
Now your trial is over and you’ve done your data analysis. But wait! Despite your best efforts, using the above tricks, your results have come back negative. What can you do now to convince people that your new drug is fantastic?
Well, firstly, you could not publish your results at all. If you have enough money, you could even run the trial again and hope that next time the results will be positive.
Alternatively, you’ve got a few more tricks up your sleeve…
Fiddle with the data…
- Ignore participants who’ve dropped out of your study. People who drop out of trials are more likely to have responded poorly to the treatment they received and to have experienced side-effects. So ignore them. Including them in your analysis could show your drug in a bad light.
- ‘Clean up’ your data. You’ll probably find some ‘outliers’ in your data. An ‘outlier’ is an observation that differs markedly from your other observations (e.g. when a particular participant responds spectacularly well or spectacularly poorly to a drug, this will skew your data). These ‘outliers’ might be helping your data (i.e. making your drug look good). If so, leave that ‘outlier’ in (even if it’s likely to be false/misleading)! But if a given ‘outlier’ is not making your drug look good, remove it!
- If your results aren’t what you hoped for, go back and analyse ‘sub-groups’ within your sample. You might find that your drug worked well within a particular sub-group, such as 30-35 year old females who own 2 cats. Who cares if it’s likely to be purely by chance that this particular group seem to have responded well and if the observation itself is clinically meaningless?
- Use a different statistical test. Some statistical tests shouldn’t be used if your data doesn’t meet certain assumptions. But who cares about using an inappropriate statistical test if it means you get the result you want!
When you’re finished fiddling with your data, choose the journal you submit your study to wisely. If you’ve used any of the tricks above, a careful reader will spot them. So you’re best off submitting your paper to an obscure journal. That way, fewer people will read it and lots of them won’t read past the abstract.
Finally, why not add a bit of spin in the discussion or conclusion of your paper!
Overstate how effective your drug is. (For example, by focusing on that obscure sub-group of your sample who responded well to your drug). Understate (or completely ignore) any side-effects or harms that you might have found. A bit of embellishment won’t go amiss!
Viola! So there we have some tricks to help you get the results you want in your quest to maximise your profits.
In all seriousness, the next time you’re reading a journal article about a clinical trial, (perhaps particularly if you see the study has been funded by a pharmaceutical company), be sceptical, be critical. Look out for any of these appalling tricks.
Disclaimer: This blog is heavily based on a chapter in Ben Goldacre’s ‘Bad Science’. I felt compelled to write this blog after reading about these shocking practices in his book. If you’re as shocked as I am by the ways in which researchers and sponsors can fiddle with their studies and data, do share this blog and read Ben Goldacre’s book for much more info.