This is the advice emerging from a study of OHS management audits conducted by the Institute for Work & Health (IWH). Led by IWH Associate Scientist Dr. Lynda Robson, the research team looked at a number of audits conducted by organizations within Ontario’s prevention system (defined as the province’s health and safety associations, Ministry of Labour, Workplace Safety and Insurance Board, Centres for Research Expertise and Institute for Work & Health).
There is a good deal of variation in the content and procedures of the audits we looked at,
says Robson. Workplaces need to consider the stage of development of their OHS management system when choosing an audit. Some audits target organizations just starting to develop their OHS management systems and others target organizations that are much further along. They also need to consider the relevance of an audit to their specific sector.
Audits play important role
OHS management audits identify strengths and weaknesses within a workplace’s OHS program in areas such as accountability, policy, hazard identification and control, training, communication and more. They do this by assessing how well the OHS program meets legislation, regulations, guidelines and established best practices. As such, audits play an important role. For example, they can:
- identify areas that need improvement in order to protect workers from injury and illness;
- ensure legislative compliance;
- benchmark OHS practices; and
- determine organizational rewards and penalties administered by regulatory and certification bodies.
Despite the importance of OHS audits, little is known about how reliable and valid they are. Robson and IWH Adjunct Scientist Dr. Philip Bigelow discovered this when they were involved in a previous project that reviewed the available research on this subject. As that 2005 review concluded, “there is little published research information on the measurement properties [i.e. reliability and validity] of OHS management audits” and what research is available “is often weak in quality.”
In this newest research, Robson and her team — with a grant from the WSIB Research Advisory Council — aimed to start filling this knowledge gap. They took a broad look at 17 audits used within Ontario’s prevention system. They also took an up-close look at five of them in particular — those determined by the researchers to be “the cream of the crop,” as Robson describes them — to find out more about audit content and procedures.
Five audit types identified
Looking at the 17 audits overall, the researchers found a lot of variety. They classified the audits into the following five types:
- legal compliance only;
- legal compliance plus some best practices beyond those incorporated in legislation;
- basic OHS management system, which assessed legal compliance, some best practices and the system framework;
- comprehensive, which assessed legal compliance and a full complement of best practices; and
- comprehensive OHS management system, which assessed legal compliance, a full complement of best practices and the system framework.
No matter what the category, all OHS audits required the auditor to review documents and interview “key informants” at the workplace (e.g. OHS manager, joint committee members, etc). Most used onsite observations to verify information gathered. Some also used them to determine the extent to which hazards were eliminated or controlled. Fewer used anonymous employee surveys to assess OHS management effectiveness.
The time required of an auditor to complete an audit, from preparation to writing the report, ranged from half a day to 15 days, with a typical length of two days. The cost to employers for an audit ranged from nothing at all to thousands of dollars, up to tens of thousands for comprehensive audits conducted at large firms.
Audit contents fairly comprehensive
The five more detailed audits chosen for closer examination fell into either the “comprehensive” or “comprehensive OHS management system” classification. These audits were compared to the contents of the Canadian Standards Association (CSA)’s Z1000-06 Occupational Health and Safety Management. This 2006 standard draws upon other well-known standards and guidelines to create a Canadian reference for designing, implementing and auditing workplace OHS management systems.
On average, the more comprehensive audits fully or partially incorporated three-quarters of the contents included in the CSA standard in their own assessments. This is a relatively high proportion,
says Robson.
Some CSA standard elements were particularly well incorporated, including worker participation, hazards and risk assessment, emergency preparedness, communication, incident analysis, and preventive and corrective action. Incorporated more weakly were elements that are characteristic of OHS system frameworks: targets/objectives, documentation, management review and internal audits.
As for administration of these audits, “variety” was again the operative word. The researchers found a range of data collection, scoring and reporting procedures. They noted differences in the recruitment, training and on-the-job assessment of auditors. As is the case in benchmarking exercises, knowledge of this variation may assist auditing organizations that are seeking to improve their audit methods,
says Robson (see below).
Inter-auditor consistency a challenge
One area of particular interest to stakeholders taking part in the study was inter-auditor consistency; in other words, ensuring that different auditors will reach similar results in similar situations. The organizations delivering the comprehensive audits pointed to this as one of their most important challenges,
says Robson. Consistency can’t be taken for granted, even with well-trained auditors. Auditing organizations are particularly interested in ensuring audit quality in this area.
Bigelow and Robson’s 2005 review pointed out that, in the very few cases where inter-auditor reliability had been examined, it was found to be “surprisingly low,” she says. This was the case, even when the audit’s content was shown to be top-notch. Inter-auditor reliability is not a large concern for audits used only for an initial assessment of OHS management systems, Robson adds. It is a concern when audits are used to measure ongoing progress in the development of an OHS management system and when they are used to certify a certain level of OHS management system quality.
Recently, Robson and Bigelow updated the 2005 research review and found that the state of the literature remains the same. For more on the 2005 click on “Occupational health and safety management audit instruments” at: www.iwh.on.ca/systematic-review-program. For more information about the current study, contact Lynda Robson at: lrobson@iwh.on.ca.
OHS management audits: Study recommendations at a glance
Based upon their review of 17 audits used within Ontario’s prevention system, the IWH research team developed “actions to consider” for auditing organizations looking for ways to improve their methods. Many had already been identified as valuable by the organizations taking part in the study. In making these recommendations, the team is not implying that each is needed by all organizations. In addition, there may be good reasons (e.g. economic constraints) for not adopting a recommendation.
The recommended actions to consider, especially when an audit is used to measure workplace performance, are:
- structuring the auditing program so that individuals do not audit organizations to which they are also providing consulting advice;
- ensuring that audits are conducted frequently enough to maintain the auditors’ skills;
- ensuring auditors get substantial practice and feedback in the field, under the guidance of expert auditors, before they begin conducting audits on their own;
- training auditors in interviewing skills and providing them with detailed interview guides;
- ensuring inter-auditor consistency in scoring evidence against audit criteria by:
- providing feedback to auditors on their scores during training
- making decision-making guidelines for auditors explicit
- measuring consistency on an ongoing basis as part of program monitoring;
- arranging field-based reviews by managers or peers to ensure ongoing learning by auditors;
- ensuring that auditors’ observations of workplace hazards and their controls play a big part in an audit’s final score;
- ensuring that the content of current audits reflects the content of the Canadian Standards Association (CSA)’s Z1000-06 standard on occupational health and management, especially in the areas of targets and objectives, internal audits, documentation and management review; and
- tying action planning closely to audit results.