Appraising the appraisal
Posted on 12th January 2018 by Dennis Neuen
Critical appraisal tools (CATs) are commonly used by students and researchers alike, as a way of judging a study’s quality. They are useful because of their structured approach, normally using categorical questions (yes/no) in standardised headings, which means that key components of study assessment are not missed e.g. internal and external validity.
The rise of FOAMed (Free Open Access Medical Education) means that learning about critical appraisal is easier than ever. Whether it’s modules or checklists, there are widespread resources all over the internet.
Somewhat ironically, an important thing to consider when using critical appraisal tools is that we need to ‘appraise’ them too. A review of CATs in 2011 showed that a comprehensive explanation of how a CAT is developed was found in only 11% of tools [1]. Similarly, only 23% and 75% of CATs had been tested for reliability and validity, respectively [1]. Whilst some questions in checklists are self-explanatory and straight-forward, there are others that are more difficult, for example ‘How precise were the results?’. Whilst many people would immediately look for the range of results and 95% confidence intervals, those with less experience in critical appraisal may need more guidance. A systematic review in 2004 showed that guidelines regarding administration were provided in only 43% of 121 published CATs [2].
Many studies have commented on the need for cross-communication between evidence-based medicine groups in order to reach a consensus and develop a suitable tool/s for critical appraisal that are valid, reliable and easily administered [2-4].
Until then, Students 4 Best Evidence have developed a non-exhaustive list of critical appraisal tools available on the internet from all over the world and we’d like you to contribute. There is no gold standard critical appraisal tool.
Are there ones that aren’t listed here? Which ones have you found useful? Let us know and we’ll update the list. (Links to all of the CATs are at the bottom of the blog)
https://jbi.global/critical-appraisal-tools
http://amstar.ca/Amstar_Checklist.php
http://thesgem.com/2014/03/make-it-so-beem-appraisal-tools/
http://www.cebm.net/critical-appraisal/
http://www.casp-uk.net/casp-tools-checklists
https://www.gla.ac.uk/researchinstitutes/healthwellbeing/research/generalpractice/ebp/checklists/
http://clinicalevidence.bmj.com/x/set/static/ebm/toolbox/665062.html
http://www.sign.ac.uk/checklists-and-notes.html
www.equator-network.org/about-us
http://bestbets.org/links/BET-CA-worksheets.php
http://www.cardiff.ac.uk/specialist-unit-for-review-evidence/resources/critical-appraisal-checklists
http://www.dartmouth.edu/~library/biomed/guides/research/ebm-resources-materials.html
References
- Crowe, M. and L. Sheppard, A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. Journal of clinical epidemiology, 2011. 64(1): p. 79-89.
- Katrak, P., et al., A systematic review of the content of critical appraisal tools. BMC medical research methodology, 2004. 4(1): p. 22.
- Quigley, J., et al., Critical Appraisal Of Real World Evidence: Review Of Recommended And Commonly Used Tools. Value in Health, 2015. 18(7): p. A684.
- Glenny, A.-M., No ‘gold standard’ critical appraisal tool for allied health research. Evidence-based dentistry, 2005. 6(4): p. 100-101.
