Comparing the validity and robustness of different statistical methods for meta-analysis of rare event data

Posted on 3rd November 2017 by Tarang Sharma

Meta-analyses of rare events or sparse data should employ different methods compared to regular data.

Assumptions for normal meta-analysis methods do not hold true in these scenarios, as there can be no events in one or both of the comparison arms. This is often the case with serious but uncommon adverse events, making it essential to get the analysis method correct as studies have shown that using a different meta-analysis method can change the final effect estimate considerably(1).

The Cochrane handbook has a separate chapter on special statistics (chapter 16), which contains a devoted section on reviews dealing with rare events (section16.9). Though it discusses the validity of the different methods available within the literature, the accompanying RevMan software only permits the use of Mantel-Haenszel odds ratio (OR) method using a 0.5 zero-cell corrections or Peto’s OR, as also acknowledged within the manual(2).

Mantel-Haenszel vs Peto odds ratio

The Mantel-Haenszel OR, using the 0.5 zero-cell corrections, have been repeatedly shown to give biased results (2,3) while the Peto´s OR has generally been considered robust when there is not a large imbalance in the total group size between the different comparison arms, and the expected effects are not large (1-3). This has meant the Peto OR has become the method of choice for most.

However, if the latter assumptions are not met, which is often the case, then the results have a higher potential to be biased. Moreover, the Peto OR model is unable to handle studies where both the arms have no events, and effectively removes them from the analysis. Therefore this further limits its applicability and ethically, patients who have been recruited to these double-zero studies, have a right that their data is included in meta-analyses (4-6).

We therefore undertook a study to identify the validity and robustness of effect estimates across different meta-analysis methods for rare binary events data, using the example of serious rare adverse events from antidepressants trials. We compared the four rare adverse outcomes of all-cause mortality, suicidality, aggressive behaviour and akathisia across the Peto method, the generalized linear mixed model (GLMM), conditional logistic regression, a Bayesian approach using Markov Chain Monte Carlo (MCMC) method and finally the beta-binomial method.

Our results, recently published online (7) in the Journal of Clinical Epidemiology, showed that though the estimates for the four outcomes did not change substantially across the different analysis methods, the Peto method underestimated the treatment harm and overestimated its precision, especially when the estimated OR deviated greatly from 1. For example, the OR for suicidality for children and adolescents was 2.39 (95% CI 1.32 to 4.33) using the Peto method, but increased to 2.64 (1.33 to 5.26) using conditional logistic regression, to 2.69 (1.19 to 6.09) using beta-binomial, to 2.73 (1.37 to 5.42) using the GLMM and finally to 2.87 (1.42 to 5.98) using the MCMC approach. Moreover when we consider absolute numbers and that these are very serious harms, the minor changes in the effect estimates make a difference.

Conclusion

Therefore, the method used for meta-analysis of rare events data influences the estimates obtained, and the exclusion of double zero-event studies can give misleading results. To ensure reduction of bias and erroneous inferences, sensitivity analyses should be performed using different methods. Other methods, in particular the beta-binomial method that was shown to be superior through simulation studies (4,7), should be considered as an appropriate alternative.

References

1. Sweeting M, Sutton A, Lambert P. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine 2004; 23(9): 1351-75.
2. Higgins JG, S (editors). Cochrane Handbook for Systematic Reviews of Interventions [updated March 2011]. Available from www.cochrane-handbook.org.: The Cochrane Collaboration; 2011, (accessed May 2016).
3. Bradburn M, Deeks J, Berlin J, Russell Localio A. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Statistics in Medicine 2007; 26(1): 53-77.
4. Kuss O. Statistical methods for meta-analyses including information from studies without any events—add nothing to nothing and succeed nevertheless. Statistics in Medicine 2015; 34(7): 1097-116.
5. Friedrich JO, Adhikari NK, Beyene J. Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data. BMC Medical Research Methodology 2007; 7(1): 1-6.
6. Keus F, Wetterslev J, Gluud C, Gooszen HG, van Laarhoven CJHM. Robustness Assessments Are Needed to Reduce Bias in Meta-Analyses That Include Zero-Event Randomized Trials. Am J Gastroenterol 2009; 104(3): 546-51.
7. Sharma T, Gøtzsche PC, Kuss O. The Yusuf-Peto method was not a robust method for meta-analyses of rare events data from antidepressant trials. Journal of Clinical Epidemiology. 2017; Aug 9. pii: S0895-4356(17)30785-0.  doi: 10.1016/j.clinepi.2017.07.006. 

Tags: