Why assess quality in a systematic review?
Posted on 3rd April 2017 by Jenni Burton
Whether it’s choosing where to eat out or what to include in the weekly shop, as diners and shoppers we are surrounded by information about quality. Detailed information, ratings and reviews are everywhere because, as consumers, we increasingly want to know more about where our food has come from, how it has been prepared and stored… Why then are we prepared to accept summaries of clinical evidence which provide no information about the quality of that evidence, despite knowing that poorly conducted research may produce misleading findings?
Methodological quality and reporting quality
When we think about ‘quality’ in research we aren’t describing the impact of the findings, instead it’s about how the study has been conducted (termed methodological quality) and how it has been described (termed reporting quality). For methodological quality, there are lots of tools out there to identify the key aspects of study design which are important. For reporting quality, the tools are checklists which help ensure studies are reported in a way that would allow the reader to understand what was done (and repeat it if needed). For both aspects the idea is to help make research more transparent so we know how it applies to our everyday practice and have enough information to critique it.
Quality in systematic reviews
As a systematic review is a synthesised summary of the available evidence, having a way to capture ‘quality’ is a vital part of interpreting the evidence. However, many published systematic reviews you read may not include any quality assessment or may reduce this bit to a single sentence, referencing a tool used and never referring to it again. However, if you read a Cochrane Review you’ll discover that the findings will all contain a careful assessment of the quality of the evidence behind any statement or recommendation made. Why? Well likely because of the potential impact of Cochrane evidence and the need for review authors to balance the numerical findings (or the statistical strength of effects) against the methodological quality.
I believe quality assessment is an essential part of every systematic review and think there is much to be done to help make it more accessible and understandable to students and researchers. My interest was sparked when I was working on the systematic review for my PhD which is all about predictors of admission to long-term care in hospitalised adults. To investigate this question I wanted to find observational study designs (studies which look at what happened to a group of people in normal clinical practice, rather than those which intervene to change what happens to them). To my surprise, there were lots of different tools in use and I was struck by how few of them have been formally evaluated. Some haven’t even been published and subjected to peer review.
‘Using quality assessment tools to critically appraise ageing research: a guide for clinicians’
The variety of tools out there, and the lack of clarity over which would be best to use, motivated me to write about quality assessment in more detail, in this paper. To help bring some clinical relevance, we used examples from ageing research to help illustrate the concepts, but the ideas are much more broadly applicable to various healthcare topics.
We looked at tools to assess methodological quality and reporting quality for randomised trials, non-randomised studies, observational studies, systematic reviews and diagnostic test accuracy studies. Our paper summarises useful resources for those interested in teaching, organising journal clubs and of course for those working on systematic reviews. This includes an introduction to the Cochrane Risk of Bias tool for randomised trials and explanation of the different types of bias.
We’ve also provided a brief introduction to the Grading of Recommendations Assessment Development and Evaluation system, known as GRADE, which is the internationally accepted way to help describe the quality of all the evidence on a topic and is often used in clinical guidelines. Finally, we highlight lots of free resources which can be accessed by those with an interest and are musts for those conducting a systematic review, including the Enhancing QUAlity and Transparency Of health Research (EQUATOR) Network and tools like GRADEpro and Review Manager.
Thanks to funding from the Medical Research Council (www.mrc.ac.uk), the paper is freely available to all, open access, available from Age and Ageing via https://academic.oup.com/ageing/article/doi/10.1093/ageing/afw223/2654237/Using-quality-assessment-tools-to-critically. We hope that sharing it will help demystify quality assessment for students, researchers and practitioners alike and help those preparing systematic reviews find a way to present quality as a key part of their findings.
