In mid-2013, WorkSafeNB adopted the Institute for Work & Health (IWH)’s eight-item Organizational Performance Metric (OPM) as a benchmark tool to measure occupational health and safety culture among employers in New Brunswick.
The decision met a long-standing need at WorkSafeNB. Responsible for both workers’ compensation and health and safety enforcement in the province of New Brunswick, WorkSafeNB engages the province’s businesses every year to assist them with their health and safety programs. Safety culture surveys are one of the tools it uses to do so, but finding a short, easy and reliable benchmarking survey was proving to be a challenge.
In 2001, WorkSafeNB began using a lengthy health and safety tool to assess the key elements of an organization’s safety culture. The tool measured perceptions of a firm’s current health and safety policy and practices, with questions covering five fundamentals and 22 topics. It had to be administered to all employees, supervisors and managers in an organization, making the process very time-consuming.
That survey was too complex and cumbersome,
says Anne Lise Albert, assistant director of Program Development and Evaluation at WorkSafeNB. It took employees away from their everyday work. As well, in workplaces where literacy was an issue, having all employees complete the surveys was problematic.
WorkSafeNB searches for shorter, easier tool
Albert’s team wanted a shorter and easier perception tool. The team did a scan across the country and found six surveys similar to what it had in mind, including the Institute’s OPM. The OPM was developed with Ontario’s prevention system partners to measure workplace health and safety performance.
We liked the work IWH had done on the OPM, and the length of the tool was attractive,
says Albert. However, we had questions about the scale used.
In other words, WorkSafeNB was unsure about the OPM’s use of a “percentage of time” scale as opposed to the “level of agreement” scale it was used to. For example, the OPM asks respondents what percentage of time workers and supervisors have the information they need to work safely, as opposed to asking their level of agreement with a statement to that effect.
So WorkSafeNB decided to create its own tool based on all six tools found. The result was an 18-item questionnaire called the Internal Responsibility System Questionnaire (IRSQ).
Two tools included in validation study
Before rolling out this new tool, Program Development and Evaluation staff at WorkSafeNB wanted to test its validity. In other words, it wanted to find out if it would measure workplace health and safety.
Because the IWH had expertise and experience validating its OPM, WorkSafeNB asked an Institute team to conduct a validation study of the IRSQ tool. The team, headed by Senior Scientist Dr. Benjamin Amick, also took the opportunity to study the OPM in the New Brunswick setting at the same time.
Both tools were sent out to about 800 employers. They were asked to have an employee, plus either a senior manager or a supervisor, fill out both the IRSQ and the OPM.
About 250 firms responded to the full request, and another 80 had only one representative complete the questionnaires. WorkSafeNB staff then provided five years of de-identified historical claims data to IWH researchers to assess how the results of each tool were associated with past injury claims.
The results showed a strong correlation between OPM scores and past workplace injury experience. That is, firms that had better OPM scores also had lower claims rates, including both lost-time and no-lost-time claims. In contrast, the results found a weak correlation between IRSQ scores and past workers’ compensation claims rates.
The two tools cover generally the same areas, with one difference being the answer scales. That’s a key difference between the two questionnaires,
says Michael Swift, a member of the IWH team. And while we would need more research to know why the IRSQ didn’t work, the answer may lie in that difference.
OPM used to benchmark and guide firms
But Keir and her team at WorkSafeNB were not focused on why the IRSQ was not validated. Our goal was to have an efficient and accurate tool,
she says. Given that the OPM had been twice validated―in Ontario and, now, in New Brunswick―WorkSafeNB decided to use the OPM as a benchmarking tool for the province’s employers.
WorkSafeNB will use the OPM with a suite of more comprehensive tools to observe changes in workplaces that are part of its Focus Firm program. The program targets firms with 40 or more employees that have a high accident count or an accident frequency greater than their industry counterparts. WorkSafeNB will use the OPM to get a baseline assessment of a participating firm’s safety culture and internal responsibility systems. It will work with these firms over the next three years to help them develop an integrated health and safety system and reduce injuries.
The OPM will be used as a high level metric,
says Albert. Her team expects to see improved OPM scores as the firms complete the questionnaire at fixed points over the course of the program.