Dataset Information

Systematic review of the use of process evaluations in knowledge translation research.

ABSTRACT: BACKGROUND:Experimental designs for evaluating knowledge translation (KT) interventions can provide strong estimates of effectiveness but offer limited insight into how the intervention worked. Consequently, process evaluations have been used to explore the causal mechanisms at work; however, there are limited standards to guide this work. This study synthesizes current evidence of KT process evaluations to provide future methodological recommendations. METHODS:Peer-reviewed search strategies were developed by a health research librarian. Studies had to be in English, published since 1996, and were not excluded based on design. Studies had to (1) be a process evaluation of a KT intervention study in primary health, (2) be a primary research study, and (3) include a licensed healthcare professional delivering or receiving the intervention. A two-step, two-person hybrid screening approach was used for study inclusion with inter-rater reliability ranging from 94 to 95%. Data on study design, data collection, theoretical influences, and approaches used to evaluate the KT intervention, analysis, and outcomes were extracted by two reviewers. Methodological quality was assessed with the Mixed Methods Appraisal Tool (MMAT). RESULTS:Of the 20,968 articles screened, 226 studies fit our inclusion criteria. The majority of process evaluations used qualitative forms of data collection (43.4%) and individual interviews as the predominant data collection method. 72.1% of studies evaluated barriers and/or facilitators to implementation. 59.7% of process evaluations were stand-alone evaluations. The timing of data collection varied widely with post-intervention data collection being the most frequent (46.0%). Only 38.1% of the studies were informed by theory. Furthermore, 38.9% of studies had MMAT scores of 50 or less indicating poor methodological quality. CONCLUSIONS:There is widespread acceptance that the generalizability of quantitative trials of KT interventions would be significantly enhanced through complementary process evaluations. However, this systematic review found that process evaluations are of mixed quality and lack theoretical guidance. Most process evaluation data collection occurred post-intervention undermining the ability to evaluate the process of implementation. Strong science and methodological guidance is needed to underpin and guide the design and execution of process evaluations in KT science. REGISTRATION:This study is not registered with PROSPERO.

SUBMITTER: Scott SD

PROVIDER: S-EPMC6836407 | biostudies-literature | 2019 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Systematic review of the use of process evaluations in knowledge translation research.

Scott Shannon D SD Rotter Thomas T Flynn Rachel R Brooks Hannah M HM Plesuk Tabatha T Bannar-Martin Katherine H KH Chambers Thane T Hartling Lisa L

Systematic reviews 20191107 1

<h4>Background</h4>Experimental designs for evaluating knowledge translation (KT) interventions can provide strong estimates of effectiveness but offer limited insight into how the intervention worked. Consequently, process evaluations have been used to explore the causal mechanisms at work; however, there are limited standards to guide this work. This study synthesizes current evidence of KT process evaluations to provide future methodological recommendations.<h4>Methods</h4>Peer-reviewed searc ...[more]

PMID: 31699136

Similar Datasets

Project description:BackgroundExperimental designs for evaluating knowledge translation (KT) interventions for professional behavior change can provide strong estimates of intervention effectiveness but offer limited insight how the intervention worked or not. Furthermore, trials provide little insight into the ways through which interventions lead to behavior change and how they are moderated by different facilitators and barriers. As a result, the ability to generalize the findings from one study to a different context, organization, or clinical problem is severely compromised. Consequently, researchers have started to explore the causal mechanisms in complementary studies (process evaluations) alongside experimental designs for evaluating KT interventions. This study focuses on improving process evaluations by synthesizing current evidence on process evaluations conducted alongside experimental designs for evaluating KT interventions.Methods/designA medical research librarian will develop and implement search strategies designed to identify evidence that is relevant to process evaluations in health research. Studies will not be excluded based on design. Included studies must contain a process evaluation component aimed at understanding or evaluating a KT intervention targeting professional behavior change. Two reviewers will perform study selection, quality assessment, and data extraction using standard forms. Disagreements will be resolved through discussion or third party adjudication. Data to be collected include study design, details about data collection approaches and types, theoretical influences, approaches to evaluate intervention dose delivered, intervention dose received, intervention fidelity, intervention reach, data analysis, and study outcomes. This study is not registered with PROSPERO.DiscussionThere is widespread acceptance that the generalizability of quantitative trials of KT interventions would be significantly enhanced to other contexts, health professional groups, and clinical conditions through complementary process evaluations alongside trials. This systematic review will serve as a 'state of the science' on methodological approaches to process evaluations and will allow us to: 1) take stock of current research approaches and 2) develop concrete recommendations for knowledge users (e.g., quality consultants and health services researchers) designing future KT process evaluations.

Project description:BackgroundKnowledge translation (KT) aims to facilitate the use of research evidence in decision making. Changes in technology have provided considerable opportunities for KT strategies to improve access and use of evidence in decision making by public health policy makers and practitioners. Despite this opportunity, there have been no reviews that have assessed the effects of digital technology-enabled KT (TEKT) in the field of public health.ObjectiveThis study aims to examine the effectiveness of digital TEKT strategies in (1) improving the capacity for evidence-based decision making by public health policy makers and practitioners, (2) changing public health policy or practice, and (3) changes in individual or population health outcomes.MethodsA search strategy was developed to identify randomized trials assessing the effectiveness of digital TEKT strategies in public health. Any primary research study with a randomized trial design was eligible. Searches for eligible studies were undertaken in multiple electronic bibliographic databases (Medical Literature Analysis and Retrieval System Online [MEDLINE], Excerpta Medica dataBASE [EMBASE], PsycINFO, Cumulative Index to Nursing and Allied Health Literature [CINAHL], and Scopus) and the reference lists of included studies. A hand search of 2 journals (Implementation Science and Journal of Medical Internet Research) and a gray literature search were also conducted. Pairs of independent review authors screened studies, assessed the risk of bias, and extracted data from relevant studies.ResultsOf the 6819 citations screened, 8 eligible randomized trials were included in the review. The studies examined the impact of digital TEKT strategies on health professionals, including nurses, child care health consultants, physiotherapists, primary health care workers, and public health practitioners. Overall, 5 of the interventions were web-training programs. The remaining 3 interventions included simulation games, access to digital resource materials and the use of tailored messaging, and a web-based registry. The findings suggest that digital TEKT interventions may be effective in improving the knowledge of public health professionals, relative to control, and may be as effective as a face-to-face KT approach. The effectiveness of digital TEKT strategies relative to a control or other digital KT interventions on measures of health professional self-efficacy to use evidence to enhance practice behavior or behavioral intention outcomes was mixed. The evidence regarding the effects on changes to health policy or practice following exposure to digital TEKT was mixed. No trials assessed the effects on individual or population-level health outcomes.ConclusionsThis review is the first to synthesize the effectiveness of digital TEKT interventions in a public health setting. Despite its potential, relatively few trials have been undertaken to investigate the impacts of digital TEKT interventions. The findings suggest that although a digital TEKT intervention may improve knowledge, the effects of such interventions on other outcomes are equivocal.

Project description:ObjectiveProcess evaluations (PEs) alongside randomised controlled trials of complex interventions are valuable because they address questions of for whom, how and why interventions had an impact. We synthesised the methods used in PEs of primary care interventions, and their main findings on implementation barriers and facilitators.DesignSystematic review using the UK Medical Research Council guidance for PE as a guide.Data sourcesAcademic databases (MEDLINE, SCOPUS, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, EMBASE and Global Health) were searched from 1998 until June 2018.Eligibility criteriaWe included PE alongside randomised controlled trials of primary care interventions which aimed to improve outcomes for patients with non-communicable diseases.Data extraction and synthesisTwo independent reviewers screened and conducted the data extraction and synthesis, with a third reviewer checking a sample for quality assurance.Results69 studies were included. There was an overall lack of consistency in how PEs were conducted and reported. The main weakness is that only 30 studies were underpinned by a clear intervention theory often facilitated by the use of existing theoretical frameworks. The main strengths were robust sampling strategies, and the triangulation of qualitative and quantitative data to understand an intervention's mechanisms. Findings were synthesised into three key themes: (1) a fundamental mismatch between what the intervention was designed to achieve and local needs; (2) the required roles and responsibilities of key actors were often not clearly understood; and (3) the health system context-factors such as governance, financing structures and workforce-if unanticipated could adversely impact implementation.ConclusionGreater consistency is needed in the reporting and the methods of PEs, in particular greater use of theoretical frameworks to inform intervention theory. More emphasis on formative research in designing interventions is needed to align the intervention with the needs of local stakeholders, and to minimise unanticipated consequences due to context-specific barriers.Prospero registration numberCRD42016035572.

Project description:BackgroundThe status of randomised controlled trials (RCTs) as the 'gold standard' for evaluating efficacy in healthcare interventions is increasingly debated among the research community, due to often insufficient consideration for implementation. Normalisation Process Theory (NPT), which focuses on the work required to embed processes into practice, offers a potentially useful framework for addressing these concerns. While the theory has been deployed in numerous RCTs to date, more work is needed to consolidate understanding of if, and how, NPT may aid implementation planning and processes within RCTs. Therefore, this review seeks to understand how NPT contributes to understanding the dynamics of implementation processes within RCTs. Specifically, this review will identify and characterise NPT operationalisation, benefits and reported challenges and limitations in RCTs.MethodsA qualitative systematic review with narrative synthesis of peer-reviewed journal articles from eight databases was conducted. Studies were eligible for inclusion if they reported sufficient detail on the use of NPT within RCTs in a healthcare domain. A pre-specified data extraction template was developed based on the research questions of this review. A narrative synthesis was performed to identify recurrent findings.ResultsSearches identified 48 articles reporting 42 studies eligible for inclusion. Findings suggest that NPT is primarily operationalised prospectively during the data collection stage, with limited sub-construct utilisation overall. NPT is beneficial in understanding implementation processes by aiding the identification and analysis of key factors, such as understanding intervention fidelity in real-world settings. Nearly three-quarters of studies failed to report the challenges and limitations of utilising NPT, though coding difficulties and data falling outside the NPT framework are most common.ConclusionsNPT appears to be a consistent and generalisable framework for explaining the dynamics of implementation processes within RCTs. However, operationalisation of the theory to its full extent is necessary to improve its use in practice, as it is currently deployed in varying capacities. Recommendations for future research include investigation of NPT alongside other frameworks, as well as earlier operationalisation and greater use of NPT sub-constructs.Trial registrationThe protocol for this systematic review was accepted for public registration on PROSPERO (registration number: CRD42022345427) on 26 July 2022.

Project description:BackgroundWithin increasingly constrained school timetables, interventions that integrate academic and health education to reduce substance use and violence may hold promise as a category of intervention that can positively affect both academic and health outcomes. There are no current systematic reviews exploring the effectiveness of such interventions or factors that affect their implementation.MethodsA total of 19 bibliographic databases and 32 websites were searched. References were also extracted from the reference lists of included studies, and experts and authors were contacted to identify relevant studies. We included reports with no restrictions on language or date. References were screened on title/abstract and those not thus excluded were screened on full report. Data extraction and appraisal followed the Critical Appraisal Skills Programme, Evidence for Policy and Practice Information and Co-ordinating Centre and Cochrane tools. Extracted process data were qualitatively meta-synthesised for common themes.ResultsSeventy-eight thousand four hundred fifty-one unique references were identified, and 62 reports were included. A total of 16 reports (reporting on 15 studies of 12 interventions) evaluated process. Key facilitators of integrated academic and health curricula were supportive senior management and alignment of the intervention with school ethos; a positive teaching environment, including positive perceptions around the ability to be flexible in the adaptation and delivery of integrated academic and health curricula; positive pre-existing student and teacher attitudes towards intervention content; and parental support of interventions, largely through reinforcement of messaging at home. Important barriers were over-burdened teachers, with little time to learn and implement integrated curricula.ConclusionSeveral useful facilitating and inhibiting factors linked to the implementation of interventions that integrate academic and health education for reduced substance use and/or violence were identified, providing tentative but insightful evidence of context-specific issues that may impact intervention success. However, overall, there is still a considerable gap in our understanding of how to achieve the successful implementation of these interventions.

Project description:BackgroundNormalization Process Theory (NPT) identifies, characterises and explains key mechanisms that promote and inhibit the implementation, embedding and integration of new health techniques, technologies and other complex interventions. A large body of literature that employs NPT to inform feasibility studies and process evaluations of complex healthcare interventions has now emerged. The aims of this review were to review this literature; to identify and characterise the uses and limits of NPT in research on the implementation and integration of healthcare interventions; and to explore NPT's contribution to understanding the dynamics of these processes.MethodsA qualitative systematic review was conducted. We searched Web of Science, Scopus and Google Scholar for articles with empirical data in peer-reviewed journals that cited either key papers presenting and developing NPT, or the NPT Online Toolkit ( www.normalizationprocess.org ). We included in the review only articles that used NPT as the primary approach to collection, analysis or reporting of data in studies of the implementation of healthcare techniques, technologies or other interventions. A structured data extraction instrument was used, and data were analysed qualitatively.ResultsSearches revealed 3322 citations. We show that after eliminating 2337 duplicates and broken or junk URLs, 985 were screened as titles and abstracts. Of these, 101 were excluded because they did not fit the inclusion criteria for the review. This left 884 articles for full-text screening. Of these, 754 did not fit the inclusion criteria for the review. This left 130 papers presenting results from 108 identifiable studies to be included in the review. NPT appears to provide researchers and practitioners with a conceptual vocabulary for rigorous studies of implementation processes. It identifies, characterises and explains empirically identifiable mechanisms that motivate and shape implementation processes. Taken together, these mean that analyses using NPT can effectively assist in the explanation of the success or failure of specific implementation projects. Ten percent of papers included critiques of some aspect of NPT, with those that did mainly focusing on its terminology. However, two studies critiqued NPT emphasis on agency, and one study critiqued NPT for its normative focus.ConclusionsThis review demonstrates that researchers found NPT useful and applied it across a wide range of interventions. It has been effectively used to aid intervention development and implementation planning as well as evaluating and understanding implementation processes themselves. In particular, NPT appears to have offered a valuable set of conceptual tools to aid understanding of implementation as a dynamic process.

Project description:BackgroundKnowledge translation (KT) platforms are organisations, initiatives and networks that focus on supporting evidence-informed policy-making at least in part about the health-system arrangements that determine whether the right programmes, services and products get to those who need them. Many descriptions and evaluations of KT platforms in low- and middle-income countries have been produced but, to date, they have not been systematically reviewed.MethodsWe identified potentially relevant studies through a search of five electronic databases and a variety of approaches to identify grey literature. We used four criteria to select eligible empirical studies. We extracted data about seven characteristics of included studies and about key findings. We used explicit criteria to assess study quality. In synthesising the findings, we gave greater attention to themes that emerged from multiple studies, higher-quality studies and different contexts.ResultsCountry was the most common jurisdictional focus of KT platforms, EVIPNet the most common name and high turnover among staff a common infrastructural feature. Evidence briefs and deliberative dialogues were the activities/outputs that were the most extensively studied and viewed as helpful, while rapid evidence services were the next most studied but only in a single jurisdiction. None of the summative evaluations used a pre-post design or a control group and, with the exception of the evaluations of the influence of briefs and dialogues on intentions to act, none of the evaluations achieved a high quality score.ConclusionsA large and growing volume of research evidence suggests that KT platforms offer promise in supporting evidence-informed policy-making in low- and middle-income countries. KT platforms should consider as next steps expanding their current, relatively limited portfolio of activities and outputs, building bridges to complementary groups, and planning for evaluations that examine 'what works' for 'what types of issues' in 'what types of contexts'.

Dataset Information

Systematic review of the use of process evaluations in knowledge translation research.

Publications

Systematic review of the use of process evaluations in knowledge translation research.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets