Funders and policy-makers want to know that their research investments are making a difference. As a result, research organizations world-wide are committing more time and resources to knowledge transfer and exchange (KTE) – the practice of putting relevant research into the hands of key decision-makers and stakeholders in a timely, accessible and useful manner.
The effectiveness of current KTE practices, however, has not been routinely or consistently evaluated. One reason for this may be the lack of valid and reliable tools for assessing the implementation and impact of KTE applications.
This systematic review sought to fill this gap. It looked across a wide variety of research fields to identify and describe approaches and instruments that can help KTE practitioners measure the impact of their work. Specifically, it was looking for tools that accurately and reliably measure how well KTE applications bring research evidence directly to practitioners (such as clinicians) and change their knowledge, attitudes and/or behaviour.
The primary objective of this review was to answer the question: Are there reliable, valid and/or useful instruments to apply in the assessment of KTE implementation and its impact?
How was the review conducted?
The review was conducted by a 13-member multidisciplinary team from Ontario, Canada. The team included KTE practitioners, KTE researchers, qualitative and quantitative researchers, and review methodologists.
The review team was supported by three decision-makers from organizations heavily engaged in KTE, as well as other representatives of key KTE stakeholders. These decision-makers and stakeholders helped shape the research question, guide the methods used in the review process, maintain the review team’s focus on practical outcomes, and recommend ways to share findings.
The review team searched for articles on knowledge transfer in nine databases from different domains: medicine, psychology, education, agriculture, library and information science, social science and business. It also hand-searched selected journals, consulted with content experts and review team members about publications in their personal libraries, and scanned the references of included studies. Once duplicates were removed, the initial search yielded 9,998 articles.
These articles were then reviewed for their relevance to the research question. The determination of relevance was guided by the following question: Does the article describe a KTE outcome or a tool to measure a KTE outcome as a result of a KTE application?
In using this question, the team agreed on two important definitions:
KTE was defined as “an iterative and dynamic process by which relevant research information is created, synthesized, disseminated and exchanged through interactive engagement with decision-makers/knowledge-users to improve outcomes, provide more effective services and products, and strengthen the use of evidence in decision-making, practice, planning and policy-making.”
A KTE application was defined as “any activity or practice in which KTE is a stated goal that is linked to specific outcomes (i.e. these activities/practices are intended to change something, be it behaviour, attitudes, capacity, decision-making, policies, programs etc.).”
After screening for relevance and removing articles published in languages other than English or French, the review team identified 346 potential articles. The review team focused primarily on qualitative articles that described explicit approaches to evaluating KTE, and on quantitative articles that contained discernible instruments to evaluate KTE. The team was looking for instruments such as a questionnaires, surveys and interviews that collected information from practitioners (i.e. the end-users of research information).
The team found 12 qualitative and 54 quantitative articles that met its criteria. Information was collected from all 66 articles about items such as KTE application (e.g. printed material, interactive workshop), type of knowledge use (e.g. conceptual, instrumental) and target audience (e.g. practitioners, policy-makers), as well as research objectives and study characteristics.
For the quantitative articles, information was also collected about the measurement properties of the instruments described. In particular, the team was looking for information on:
- Validity: Does the instrument measure what it says it measures?
- Reliability: Does the instrument consistently measure what it says it measures, such that its measurements can be reproduced?
- Responsiveness: Is the instrument able to detect change (e.g. in knowledge, attitudes, behaviour) resulting from a KTE application.
What were the main findings?
Most of the articles were about KTE activities in the health-care field (89%), and the audience for these KTE activities was largely practitioners (92%), such as clinicians. The articles described and evaluated a variety of KTE applications. The most common were printed materials such as booklets or guideline checklists (in 65% of articles), interactive in-person workshops (52%), electronic materials (21%), opinion leader (18%) and train-the-trainer (17%). The aims of the KTE applications were largely conceptual (to change understanding or attitudes, 80%) and instrumental (to change behaviour, 71%).
Overall, the review team found few well developed approaches or instruments to evaluate KTE implementation or its impact.
Although the 12 qualitative articles provided good descriptions of process, context and impact, as well as barriers and facilitators in the evaluation of KTE applications, their overall quality was low. However, the four higher quality articles among this group indicated that well-designed case studies and cross-case comparisons could prove useful to the evaluation of KTE applications in future.
The 54 quantitative articles described a variety of instruments for evaluating KTE activities, many of which were context-specific, developed by researchers for their particular studies. Most articles did not clearly report instrument measurement properties, and those that did were often lacking detail. As well, none of the articles provided information to show the instruments were able to measure meaningful changes in knowledge, attitude and/or behaviours among practitioners.
That said, the review team did find 16 quantitative articles describing instruments that showed promise as potentially useful tools in the evaluation of KTE activities. They were considered promising because they demonstrated some signs of validity and reliability, as well as the potential to assess change.
Because these instruments are not well-established or widely known by name, the best way to learn more about them is to read the articles in which they are described. The full references for these articles, as well as brief descriptions of the evaluation instruments they describe, can be found in a companion Sharing Best Evidence called Promising KTE evaluation tools.
The review team found another three articles describing instruments under development that also showed promise.
What are the implications?
Working with stakeholders, the review team developed the following messages based on the review’s findings. These messages are directed to KTE researchers and KTE practitioners.
KTE researchers and practitioners should:
- Be aware of, and consider, the evaluation instruments found in this review when planning an evaluation study. When considering an instrument,always look for a clear presentation of the measurement properties.
- Select or construct well-developed instruments for evaluation. The minimum measurement standards for a KTE evaluation instrument should be demonstrated validity and reliability.
- Consider developing instruments that can be used in various contexts. Remember to separate instrument implementation, where context is very important, from instrument development, which can be theory-based and independent of context.
- Systematically develop and use evaluation instruments with known measurement properties in order to advance KTE evaluation. Clearly demonstrate that the instrument can measure meaningful change among users of research evidence.
KTE researchers should:
- Involve KTE practitioners in the development and evaluation of KTE instruments in order to increase KTE practitioners’ knowledge of measurement and evaluation and researchers’ appreciation of the practical demands of doing KTE.
- Consider developing reporting guidelines to ensure KTE research articles and scientific reports are rigorously written, following standards of scientific reporting. The use of a consistent format, as well as consistent terms and definitions, would allow for better communication, information retrieval and, ultimately, assessment of KTE practices.
- Continue to conduct high calibre qualitative studies, such as case studies and cross-case analyses, which provide rich detail about the process, context and evaluation of KTE approaches.
- Use or develop instruments that are shown to measure meaningful change to ensure evaluation will be fairly assessed.
The review team also offered words of advice for KTE practitioners (see box below).
Conclusions
The goal of this review was to find out if reliable and valid instruments are available to assess the implementation and impact of KTE activities. It appears that, at this time, they are not. The review team found few well-developed instruments to evaluate KTE implementation and its impact.
The team was surprised by the lack of theory-based instruments; most instruments were developed for specific KTE activities. Furthermore, these context-specific instruments did not, as a rule, clearly report measurement properties.
Nonetheless, some instruments do appear to be promising for the evaluation of KTE activities and practices. In addition, some instruments under development look promising.
The review team strongly encourages that KTE practitioners and researchers continue to work together to develop instruments to evaluate KTE activities, with a focus on establishing sound measurement properties. As well, in future studies reporting on KTE evaluation tools, the review team also stresses the importance of clearly presenting their measurement properties. This is necessary if the KTE field is to move forward in evaluating and developing theory-based instruments that can add to the evidence base.