Non-academic OHS sources enrich systematic reviews

IWH review team also finds workload a challenge when including grey literature in systematic reviews

Published: February 28, 2014

In the field of workplace health, good evidence quite often exists outside scientific journals. Occupational health and safety practitioners exchange best practices via trade publications and conferences more often than they do peer-reviewed articles.

In recognition of this type of evidence—known as “grey literature”—an Institute for Work & Health (IWH) systematic review team cast a wide net when it looked for evidence on the successful implementation of participatory ergonomics (PE) programs. Grey literature refers to a broad range of sources outside traditional research publications, from conference papers and dissertations to blogs and wikis.

Despite the challenges encountered, the review team found benefit in stepping beyond the confines of a traditional systematic review to include grey literature, says Quenby Mahood, IWH’s manager of library sciences.

The main challenge was the time it took. The work needed at every step was considerable, says Mahood. Yet, in the end, many of the grey-literature sources we found met our content and quality criteria, and enriched the overall findings.

Mahood wrote about the review team’s approach in Research Synthesis Method, in an article that first appeared online in December 2013 (doi:10.1002/jrsm.1106).

Providing advice on including grey literature

Researchers doing systematic reviews are meticulous about having clear, transparent and reproducible methods throughout: on how to search the literature, how to decide what studies to include and how to grade them for quality.

A body of literature has developed on how to use peer-reviewed articles in systematic reviews. However, not much advice exists on including grey literature. Part of our motivation in writing this paper was to share in detail our experience, which may help others facing similar challenges, says Mahood.

One challenge facing the review team was how to design a search strategy to turn up as much relevant information as possible without overwhelming the project. The team consulted with stakeholders and narrowed their search to include magazine articles, trade publication articles, academic dissertations, institutional reports, consultant reports, book chapters and conference proceedings. Stakeholders identified these as the types of sources where PE information would most likely appear.

That was one of the key messages we heard from stakeholders—that to not include grey literature would mean losing an important source of information, she adds.

The team decided not to include internet searches (such as a Google search) in its strategy. It would have been too difficult to deal with the large number of search results. Plus, the team was concerned that it would not be possible to reproduce the same search results in the future.

In the end, 52 of more than 2,100 identified articles made it to the last step of data extraction. They were of sufficient quality that reviewers felt confident in using them in their final messages about what makes PE programs successful. Of these, 19 were grey-literature sources.

Including grey literature provided important contextual information that you might not find in peer-reviewed articles, such as detailed information on processes around participatory ergonomics, says Dwayne Van Eerd, principal investigator of the systematic review. It really helped round out our understanding of this method of preventing musculoskeletal disorders.

For information on the PE systematic review, go to: