by Louise Tanner
A network of researchers from health and other disciplines, who are based at, or affiliated with, Newcastle University, have initiated the Systematic Reviews Group, which meets on a monthly basis to discuss topics relevant to undertaking systematic reviews. The group has been running for several years; the role of group organiser has flitted between staff and student members of IHS, until the role was taken over by the Evidence Synthesis Team to provide a forum at which advice pertaining to systematic review methods could be provided to groups of people rather than responding to individual emails, many of which that raised the same questions. The format of each meeting involves a presentation from a member of staff or a student with experience of using a particular systematic review method, followed by questions and discussion between group members.
The focus of recent meetings has been on quality appraisal tools. Quality appraisal is the process of systematically assessing the robustness of scientific research and assessing the extent to which results can be used to answer a particular question. A variety of tools are available for quality appraising different types of study and the choice of which tool(s) to implement is often difficult and depends on several factors: robustness (i.e. sensitivity, validity and reliability) of each too, resource availability (some tools are time consuming to implement than others), and which aspects of quality are relevant to individual reviews.
In a recent meeting, quality appraisal tools for quantitative epidemiological studies were discussed. The public health context of quantitative epidemiological studies led to a shortlisting of two different approaches to quality appraising: the Cochrane risk of bias tools (ROB 2.0 revised for randomised controlled trials (RCTs) and ROBINS-I for non-randomized quantitative studies), versus The Effective Public Health Practice Project Quality assessment tool for quantitative studies (EPHPP). The advantages and disadvantages of each approach were compared during the group discussion, focusing on the issues that each tool assesses.
The Cochrane risk of bias tools for quantitative studies emphasize issues affecting internal validity, which is the extent to which we can infer that the effects observed in a study are attributable exposures or interventions of interest rather than other factors (i.e. confounding variables or sources of bias). Internal validity provides information regarding the effectiveness of a public health intervention, whereas external validity (generalizability) examines whether or not the intervention is likely to be effective in settings, and with populations, which are different to those in which the intervention was implemented in the primary studies. The EPHPP includes domains for assessing both internal and external validity of evidence from individual studies; it is also less time consuming to implement than the Cochrane ROB 2.0 However, the EPHPP is a less sensitive tool as it contains fewer domains of appraisal than its Cochrane counterparts. The outcome of the group discussion regarding these tools was that the decision on which tool to implement depends on the intended outcome of each systematic review, i.e. whether it aims to assess the efficacy of a public health intervention, in which case the Cochrane ROB tools are most appropriate, or to inform the development of an intervention that can be rolled out across different population sub-groups and in different contexts, in which case the EPHPP can provide helpful information. It was also concluded that in some cases it is necessary to adapt the quality appraisal tool, for instance by combining domains from different tools, to address multiple issues (i.e. to assess intervention efficacy and generalizability).
Future Systematic Reviews Group sessions that are scheduled for the current academic year include: ‘conducting an umbrella review’, ‘introduction to meta-analysis’ and ‘introduction to network meta-analysis’. Anyone who is interested in joining our mailing list to find out about dates and times for scheduled sessions (which are normally held on the last Thursday of each month from 12:30 – 13:30pm) should contact Louise Tanner at louise.tanner@newcastle.ac.uk.