Getting feedback on early results
In this series, Research 101, we are taking you behind the scenes of a research project at the Institute for Work & Health (IWH), from start to finish.
In Part 1, we introduced you to the lead researcher, IWH Scientist Dr. Peter Smith, who formulated the research question: Why, over a 14-year period, have lost-time claims in Ontario decreased by more than 40 per cent while no-lost-time-claims have decreased by only four per cent? In Part 2, we learned how the research team overcame a number of unanticipated roadblocks during data collection.
Peers help confirm findings
By the fall of 2008, three-quarters of a year into the project, Smith is starting to analyze no-lost-time trends based on the data available. He is also ready to share early findings with fellow researchers and non-academic stakeholders.
That’s because Smith, like most scientists, wants to make sure the information he produces is meaningful and reflects what is actually happening in “the real world” — in his case, the world of OHS and workers’ compensation. He wants to explain, not just describe. That’s where feedback becomes important.
I wanted to find out if I was on the right track, if there was another way of looking at the information, or if there were leaps in my logic, says Smith.
I also wanted help developing a coherent story from this information.
Smith gets important feedback from two sources: participants at the International Forum on Disability Management in Berlin and his own research team at IWH.
Based on the information already collected for the study, Smith is ready to start analyzing why no-lost-time claims (NLTCs) have not declined at the same rate as lost-time claims (LTCs). Smith is using claims data extracted from the WSIB’s administrative records and combining it with labour force participation data from Statistics Canada. This allows him to look at no-lost-time claims across age, gender and industry group over the last 15 years.
Administrative data refers to information regularly coded and stored by the Workplace Safety and Insurance Board (WSIB) in the course of providing benefits to injured or disabled workers in Ontario. With respect to NLTCs, that means information on gender, age, industry sector, workplace size and the health-care costs associated with each claim — but not the more detailed information on nature of injury, body part injured and more that is also regularly coded for LTCs.
Conference feedback leads to modifications
In September 2008, Smith travels to Berlin to present initial findings at IFDM 2008, the 4th International Forum on Disability Management. Smith sees IFDM as the most appropriate conference to do this because it attracts policy-makers, clinicians, injured workers and service providers, as well as other academic researchers. As a result of his discussions with conference participants, Smith does indeed decide to modify his analysis of the administrative data.
Smith’s study includes examining the costs and percentage of no-lost-time claims requiring health care. Initially, Smith is only looking at the total amount of health care required in the first year post-injury. However, following Berlin, he decides to also look at the pattern of health care costs during the first year. Why? Because needing a small amount of health care over a long period of time (first year) versus needing a large amount of health care over a short period of time (first month) might tell him more about differences in the severity of claims, over time, among workers filing NLTCs.
Research team is brought together
Smith reworks his initial findings. Soon after, in October 2008, he brings the full research team together to go over the results. Up until this point, Smith, as the principal investigator and lead author, has done most of the analysis of the claims data.
You can’t get five busy researchers together every time a decision has to be made about what the data means or what the next steps should be, explains Smith.
Indeed, during the first half of a study, the lead researcher usually works alone, or with a research associate, to generate and understand the data. This is then shared with the rest of the team.
Smith wants specific feedback from the research team. He wants it to correct any lapses in deciphering the results, identify what information is most useful, and suggest what journal papers, if any, can be written based on the administrative data only. He also wants it to help develop a “coherent story” based on the emerging information.
I wanted to know if the information was giving rise to themes or angles that give shape to the information in a comprehensive way, explains Smith.
I wanted to know what stories should be told.
Each of the other four IWH colleagues on the team — President and Senior Scientist Dr. Cam Mustard, Senior Scientist Dr. Sheilah Hogg-Johnson, Scientist Dr. Emile Tompa and Data and Information Systems Research Associate Marjan Vidmar — brings an expertise and perspective that will help answer these questions and determine if Smith is on the right track.
Evidence supports three hypotheses
To ensure they are all on the same page, Smith reiterates to the team what a no-lost-time claim is. A NLTC is based on an injury that requires health care but not an absence from work other than the day of injury. It includes instances in which an injured worker performs modified work at the same pay for up to seven calendar days (after which it does become a lost-time claim).
Smith outlines to the team the three trains of thought he is pursuing about why no-lost-time claims have not declined at the same rate as lost-time claims. He also suggests how the statistics might support each one.
The primary prevention hypothesis suggests that NLTCs are not declining at the same rate as LTCs because improved primary prevention practices (e.g. training, installing machine guards, implementing engineering controls) are resulting in fewer severe injuries requiring time off work. Therefore, as the number of LTCs decline, NLTCs remain stable.
Based on this hypothesis, you might expect that health-care costs associated with no-lost-time claims will remain relatively stable over time. In addition, you might expect a higher ratio of no-lost-time to lost-time claims in the more dangerous industries that are usually targeted by primary prevention initiatives.
The accommodation hypothesis suggests that NLTCs are not declining at the same rate not because prevention practices are leading to fewer severe injuries, but because improved accommodation practices are allowing injured workers, even those with serious injuries, to return to work the next day.
Under this hypothesis, you might expect increasing health-care costs and, potentially, health care for a longer duration. You might also expect a higher ratio of NLTCs to LTCs in larger workplaces, as they are more likely to have the capacity to move injured workers into different, less demanding occupations while they recover from injury.
The claims management hypothesis suggests that NLTCs are not declining at the same rate because workplaces are encouraging workers to submit what should be lost-time claims as no-lost-time claims so employers can improve their experience rating and reduce their workers’ compensation premium payments.
Under this hypothesis, you might expect a higher ratio of NLTCs to LTCs among more precarious employees (e.g. younger workers), who may be more easily influenced to report a NLTC even when an injury has resulted in time away from work. You might also expect a decline in average health-care costs because the health care, along with the injury, is not reported to the WSIB.
Smith shows the research team how evidence is available to support each of these three hypotheses, to differing extents. The researchers discuss these findings and agree that all three conclusions, at this point, are still on the table.
I had hoped the administrative data would point to one of the theories in particular, says Smith.
Instead, it’s showing all three are possible. And it could be that all three are happening, so at this point, it’s hard to show if the change in no-lost-time claim rates is due to primary or secondary interventions, or to an emphasis on claims management.
Journal paper to be written by summer
Now, Smith is preparing for an “open” plenary at the IWH, to which both IWH and non-IWH scientists and stakeholders are invited to provide further feedback on his results to date.
It’s good to get feedback from others outside IWH who are interested in this research, he says. After the plenary, Smith will begin writing a research paper on findings from the administrative data, which he expects to submit to a journal during the summer.
In the meantime, more detailed information on NLTCs will soon be added to the mix. Linda Kacur, the person seconded from the WSIB to code the 10,000 NLTCs, will finish up her work by the end of March 2009 (see Part 2). Smith is counting on this information to tell him more about what the changes in no-lost-time claim rates really mean, and perhaps indicate more strongly which of the hypotheses is at play.
Once this information is available, Smith will go through a process similar to the one he just went through with the administrative data: prepare preliminary results, present them to IWH and non-IWH researchers and stakeholders, use their feedback to help give shape to the story being told by the evidence, and write the results for publication.
In Part 4, we’ll look at the final stages of an IWH research project. This includes submitting papers and getting the research into the hands of stakeholders who are in a position to improve OHS and workers’ compensation systems based on research findings.
Source: At Work, Issue 56, Spring 2009: Institute for Work & Health, Toronto