How we do systematic reviews

The Institute for Work & Health (IWH) is known globally for its in-house expertise in synthesizing the best available evidence on what works to protect and improve the safety and health of working people. We are especially known for our expertise in conducting systematic reviews. IWH began conducting systematic reviews and other types of knowledge synthesis in 1994. This activity expanded considerably in 1996 when we began housing Cochrane Back and Neck, one of over 50 international systematic review groups in Cochrane  Then in 2005, IWH created its Systematic Review Program—a program of research dedicated to conducting systematic reviews that answer important questions relevant to stakeholders in workplace health and safety.

Our six-step process

A systematic review is a type of research study. It aims to find an answer to a specific research question by summarizing the results across existing scientific studies. This provides evidence that is stronger and more certain than the results of individual studies.

When conducting systematic reviews, our aim at the Institute is to conduct a review of the literature that is comprehensive, systematic, rigorous and transparent. We use a process modelled on, and adapted from, the process used by Cochrane. It includes six steps, which are described below.

One of the innovative aspects of our systematic review process is our engagement of knowledge users and collaborators. They are involved throughout the review, beginning with the refinement of the research question(s) through to the extraction and communication of the messages (see figure below).

Figure illustrating the 6 steps of iwh systematic review
IWH systematic review steps and stakeholder engagement (Irvin, Van Eerd et al, 2010; Keown, Van Eerd & Irvin, 2008)

Step 1: Develop the question

To initiate a review, it is important to develop a well informed and constructed research question, optimally following a PICO (Population, Intervention, Comparator and Outcome) framework. In other words, the question clearly indicates which population of individuals is being studied, what is being done to them, compared to what, and what is the outcome of interest. For example, the following is a systematic review question that follows the PICO framework:

Population = persons with low-back pain
Intervention = alternative therapies
Comparison = usual care
Outcome = improved health

Based on this, the research question becomes: Do alternative therapies improve the health of patients suffering from back pain?

Step 2: Conduct a literature search

To find sources of evidence, we work with an information specialist to design a comprehensive search strategy that is based on the PICO parameters and is agreed upon by the project team.

Step 3: Identify relevant articles

Next, we filter the studies retrieved in the search through the use of predefined inclusion/exclusion criteria, eliminating those studies that are not deemed directly relevant to the question being asked.

Step 4: Appraise the quality of articles

Relevant studies are then appraised for their methodological rigour (i.e. were they conducted properly), using predefined criteria.

Step 5: Extract the data

The research team then summarizes the key contents of each retained study that meets the previous criteria, focusing on elements that will help answer the research question.

Step 6: Synthesize the evidence

The relevant evidence is then synthesized, which involves an assessment of the quality, quantity and consistency (based on a “best evidence synthesis” approach developed by Robert Slavin: https://doi.org/10.3102/0013189X015009005). Using this approach, and working with stakeholders, we operationalize this best evidence synthesis approach by ranking the level of evidence as strong, moderate, limited, mixed and insufficient, according to the criteria outlined in the chart below.

As the chart shows, when the level of evidence is strong, we feel we can provide practice recommendations. When the level of evidence is moderate, we provide practice considerations. When the level of evidence is lower than moderate, we feel we cannot provide directions for practice based on the scientific literature at that time.

Applying our evidence synthesis algorithm
Level of Evidence Minimum Quality* Minimum Quantity Consistency Strength of Message
Strong High (H) 3 3H agree; If 3+ studies, ≥ 3/4  of the M & H agree Recommendations
Moderate Medium (M) 2H or 2M & 1H 2H agree or 2M & 1H agree; If 3+, ≥ 2/3 of the M & H agree Practice considerations
Limited Medium (M) 1H or 2M or 1M & 1H 2 (M and/or H) agree; If 2+, >1/2 of the M & H agree Not enough evidence to make recommendations or practice considerations
Mixed Medium (M) 2 Findings are contradictory
Insufficient Medium quality studies that do not meet the above criteria
*High = Score >85% in quality assessment; Medium = Score ranges from 50-85% in quality assessment