Is it worth it? Determining the costs and benefits of workplace interventions

Published: August 10, 2006

Consider this hypothetical scenario at a company that produces telecommunications equipment. Several managers notice that workers on a particular assembly line are taking sick leave more often than other workers. Upon investigating, they learn that the assembly process causes pain in workers’ wrists, arms and shoulders. The managers identify two courses of action:

Bring in an ergonomics consultant to assess the situation and implement an injury prevention program at an estimated cost of $18,000.
Rotate workers on different assembly lines to give them a “break” at an estimated cost of $6,000 for retraining.

Which option should the managers choose? For a business that wants to protect its employees and is also concerned about the bottom line, option two might seem more appealing. However, this option may not be the best one. It might not be as effective in reducing injury and sick leave as option one. In addition, there may be “hidden costs” that haven’t been considered, such as the costs of rescheduling shifts and reduction in product quality. Option one might produce better health outcomes, but are these benefits large enough to outweigh the incremental costs? Possibly, the status quo is better than either option one or two when all costs and all benefits of each option are compared. To make a better decision, the company needs to identify all the relevant alternatives that are available, and assess all the associated costs and benefits. The type of study that considers these issues is called an economic evaluation.

Economic evaluations are important in any area where you need to make a decision about resource allocation, so that you can answer the question ‘Is it worth implementing this alternative rather than another one?’ says Emile Tompa, a Scientist and economist at the Institute for Work & Health. Many workplace interventions have been evaluated for their effectiveness since the 1980s, but very few of these have included an economic evaluation. The few that do are generally poor in quality, as Institute scientists discovered while conducting a systematic review of workplace intervention studies with economic evaluations.

Review finds low quality in workplace economic evaluations

Institute scientists are conducting a systematic review to assess the quality and quantity of evidence on economic evaluations of interventions for occupational health and safety (OHS). A pilot review completed in 2005 found only a handful of studies, and the quality was generally quite low.

Very few occupational health and safety studies have also undertaken an economic analysis, says Emile Tompa, a Scientist and economist at the Institute for Work & Health. After systematically screening a subset of studies in this field, researchers found only 23 relevant studies. Just 11 of the 23 were full economic evaluations that considered both the costs and consequences of the intervention. The rest were partial evaluations that only considered the savings of a given intervention, and not the costs. The interventions included ergonomics, participatory ergonomics and return-to-work interventions in settings as diverse as offices, warehouses, hospitals and manufacturing companies.

A lot of the studies were not undertaken by economists, says Claire de Oliveira, a PhD candidate in economics who is involved in the review. This may explain why the studies lacked the depth of a full economic evaluation. In addition, in most studies researchers did not consider multiple viewpoints or question the figures provided to do the analysis. Also, few of the studies lasted long enough to see if the programs could be sustained. There was also not enough information on the context and on the way things were measured. This made it difficult for researchers to evaluate the quality of the studies, or for a reader to be able to assess the intervention’s applicability in other settings.

As with other systematic reviews, researchers met with stakeholders early in the planning stages to help refine the review questions, search strategy and analysis plan. Stakeholders included individuals from the Workplace Safety & Insurance Board, Workers Health & Safety Centre, Ministry of Labour, Ontario Service Safety Alliance, the University of Waterloo and Dofasco. We want to ensure the product is something that they will use, says de Oliveira. Once the review is complete, likely by the fall, stakeholders will meet again to discuss the findings. In addition to preparing a report summarizing their findings, the researchers will produce a document outlining best practices in the use of economic evaluation methods in OHS intervention studies.

The Institute is making efforts to advance the use of economic evaluations in occupational health and safety (OHS) studies to address the need for higher standards in this area. One major project is to develop a book that provides guidance on how to conduct, commission and assess economic evaluations in OHS studies. To this end, the Institute hosted a two-day international workshop in April 2006, which was attended by economists and others with relevant expertise. The idea behind the workshop was to give guidance on how best to do economic analyses of OHS interventions, says Dr. Tony Culyer, the Institute’s Chief Scientist. The content of each chapter was discussed at the workshop. Within each chapter, invited authors were asked not only to identify the main barriers to producing high-quality, useful studies, but also to propose solutions. The chapters cover topics such as the perspective from which the study is done, strengthening the workplace-researcher relationship, understanding international differences in labour legislation and policy, and choosing the type of economic analysis. We want the book to be very useful for practitioners, says Tompa. The target audience includes OHS practitioners, occupational health clinicians, workplace researchers, applied economists, as well as policy-makers at workers’ compensation boards and in ministries of labour. Readers will need to have some familiarity with economics to appreciate the book’s contents, but they don’t need to be economists, he says.

What is an economic evaluation?

An economic evaluation is a type of study that can help a decision-maker choose how to allocate resources, such as money, people, time or equipment. It identifies all the options a person might reasonably select. It then compares each one in terms of all the relevant costs and consequences.

There are several types of economic evaluations, which try to answer the questions described below. The examples given are from the health-care field, which is further developed in economic evaluation.

Cost Minimization Analysis – What is the least costly way to get a given result? This approach focuses on costs. It assumes that the consequences are similar for the different options being evaluated. One example from the health-care field is comparing the costs of performing a surgery on an inpatient or outpatient basis.

Cost-Benefit Analysis – What are the costs and benefits associated with different options? This approach measures consequences in monetary terms. For example, this type of analysis might be used to look at the cost of having a universal chicken pox immunization program versus not having one.

Cost-Effectiveness Analysis – What are the costs and health outcomes associated with different options? This approach measures consequences in units that measure natural benefits, such as life-years saved and reduction in disability days. For example, a cost-effectiveness study might compare the costs and reduction in disability days associated with two drugs to treat asthma.

Cost-Utility Analysis – What are the costs and gains associated with different options? This approach measures consequences in terms of improved health or satisfaction gained (utility) /value relative to other options (value weighted units). The most common type is the Quality Adjusted Life Years (QALYs), which combines the number of years and quality of life gained for each option. For instance, suppose a depression treatment study was comparing drug therapy alone versus a combination of drug therapy and counselling. The researcher might consider measuring consequences – such as improvements in patients’ quality of life – from each option in QALYs.

Deciding which type of analysis to do depends on the question that needs to be answered. It also depends on the perspective taken, such as the worker, company or society. The data and resources that are available to conduct the evaluation are also a factor. If there are good ways to measure outcomes in terms of natural units, but they are difficult to convert to dollars, then the researcher might consider a cost-effectiveness analysis.

The book aims to set a standard in terms of good practice that will be useful for conducting economic evaluations or assessing whether a study’s results are applicable to other settings. Several methods books already exist for the evaluation of health-care interventions, which is further advanced in the use of economic evaluation methods than the OHS arena. Ideally economic evaluations should be part of every workplace intervention study, says Tompa. Someone with economics expertise needs to be involved at the outset of studies. Questions about the costs and consequences of an intervention should be built into studies of effectiveness at the front end. For example, the Ontario government is investing $60 million to install patient lifts in 500 hospitals and nursing homes across the province. Researchers are studying the effectiveness of these lifts in preventing work injuries among healthcare workers. An economic evaluation is being conducted at the same time to determine the costs and consequences associated with the lifts.

An economic evaluation can be done from different perspectives. For example, in the case of the telecommunications company, the evaluation could consider the workers’ perspective by focusing on loss of income, out-of-pocket costs, and decline in health. Or it might look at the workers’ compensation perspective by focusing on claims reductions and related savings in administration and wage-replacement. While it may seem surprising that a book on this topic doesn’t already exist, there are several reasons for this. First, the workplace arena is complex. It is difficult to assess the full range of costs and consequences, because there are a number of parties who bear the costs. In addition, collecting good data at workplaces can be challenging, particularly because of the amount of time it takes to conduct surveys. Finally, differing and sometimes conflicting priorities make it difficult to quantify all costs and consequences.

In addition, economists with an interest in OHS are relatively few and far flung, Culyer points out. With the presence of several economists at the Institute, including Culyer and Tompa, as well as scientists with expertise in workplace interventions, We have the beginning of a critical mass, he says. The deadline for completion of the book is spring 2007. Other activities to promote workplace-based economic evaluations may follow. One possibility is to develop courses in economic evaluation for those in the occupational health and safety field, similar to those offered in the health-care field, says Tompa.

Is it worth it? Determining the costs and benefits of workplace interventions

Stay Connected

Institute for Work & Health

Contact Us