Dataset Information

Development and Testing of a Novel Measure to Assess Fidelity of Implementation: Example of the Mini-AFTERc Intervention.

ABSTRACT:

Background

Fidelity of implementation (FOI) reflects whether an intervention was implemented in clinical practice according to the originally developed manual and is a key aspect in understanding intervention effectiveness. To illustrate this process of developing a fidelity measure, this study uses the Mini-AFTERc, a brief psychological intervention aimed at managing breast cancer patients' fear of cancer recurrence, as an example.

Objectives

To illustrate the development of an FOI measure through (1) applying this process to the Mini-AFTERc intervention, by including the design of a scoring system and rating criteria; (2) content validating the FOI measure using thematic framework analysis as a qualitative approach; (3) testing consistency of the FOI measure using interrater reliability.

Methods

The FOI measure was developed, its scoring system modified and the rating criteria defined. Thematic framework analysis was conducted to content validate the FOI measure using nine intervention discussions between four specialist cancer nurses and four breast cancer patients, and one simulated breast cancer patient. Intraclass-correlation was conducted to assess interrater reliability.

Results

The qualitative findings suggested that the Mini-AFTERc FOI measure has content validity as it was able to measure all five components of the Mini-AFTERc intervention. The interrater reliability suggested a moderate to excellent degree of reliability among three raters, r _ICC = 0.84, 95% CI [0.51, 0.96].

Conclusion

The study has illustrated the steps that an FOI measure can be developed through a systematic approach applied to the Mini-AFTERc intervention. The FOI measure was found to have content validity and was consistently applied, independently, by three researchers familiar with the Mini-AFTERc intervention. Future studies should determine whether similar levels of interrater reliability can be obtained by distributing written and/or video instructions to researchers who are unfamiliar with the FOI measure, using a larger sample. Employing developed and validated FOI measures such as the one presented for the Mini-AFTERc would facilitate implementation of interventions in the FCR field in clinical practice as intended.

Clinical trial registration

www.ClinicalTrials.gov, identifier: NCT03763825.

SUBMITTER: Brandt NG

PROVIDER: S-EPMC7723987 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development and Testing of a Novel Measure to Assess Fidelity of Implementation: Example of the Mini-AFTERc Intervention.

Brandt Nathalie Georgia NG McHale Calum Thomas CT Humphris Gerald Michael GM

Frontiers in psychology 20201125

<h4>Background</h4>Fidelity of implementation (FOI) reflects whether an intervention was implemented in clinical practice according to the originally developed manual and is a key aspect in understanding intervention effectiveness. To illustrate this process of developing a fidelity measure, this study uses the Mini-AFTERc, a brief psychological intervention aimed at managing breast cancer patients' fear of cancer recurrence, as an example.<h4>Objectives</h4>To illustrate the development of an F ...[more]

PMID: 33324299

Similar Datasets

Project description:BackgroundAlong with the increasing prevalence of chronic illness has been an increase in interventions, such as nurse case management programs, to improve outcomes for patients with chronic illness. Evidence supports the effectiveness of such interventions in reducing patient morbidity, mortality, and resource utilization, but other studies have produced equivocal results. Often, little is known about how implementation of an intervention actually occurs in clinical practice. While studies often assume that interventions are used in clinical practice exactly as originally designed, this may not be the case. Thus, fidelity of an intervention's implementation reflects how an intervention is, or is not, used in clinical practice and is an important factor in understanding intervention effectiveness and in replicating the intervention in dissemination efforts. The purpose of this paper is to contribute to the understanding of implementation science by (a) proposing a methodology for measuring fidelity of implementation (FOI) and (b) testing the measure by examining the association between FOI and intervention effectiveness.MethodsWe define and measure FOI based on organizational members' level of commitment to using the distinct components that make up an intervention as they were designed. Semistructured interviews were conducted among 18 organizational members in four medical centers, and the interviews were analyzed qualitatively to assess three dimensions of commitment to use--satisfaction, consistency, and quality--and to develop an overall rating of FOI. Mixed methods were used to explore the association between FOI and intervention effectiveness (inpatient resource utilization and mortality).ResultsPredictive validity of the FOI measure was supported based on the statistical significance of FOI as a predictor of intervention effectiveness. The strongest relationship between FOI and intervention effectiveness was found when an alternative measure of FOI was utilized based on individual intervention components that had the greatest variation across medical centers.ConclusionsIn addition to contextual factors, implementation research needs to consider FOI as an important factor in influencing intervention effectiveness. Our proposed methodology offers a systematic means for understanding organizational members' use of distinct intervention components, assessing the reasons for variation in use across components and organizations, and evaluating the impact of FOI on intervention effectiveness.

Project description:BackgroundThe mini-CEX is a valid and reliable method to assess the clinical competencies of trainees. Its data could be useful for educators to redesign curriculum as a process of quality improvement. The aim of this study was to evaluate a mini-CEX assessment program in our internal medicine residency training. We investigated the impact of mini-CEX workshops as a faculty development program on the acquisition of cognitive knowledge and the difference of practice behaviors among faculty members used the mini-CEX to assess residents' performance at work.MethodsWe designed an observational, two-phase study. In the faculty development program, we started a mini-CEX workshop for trainers in 2010, and the short-term outcome of the program was evaluated by comparing the pretest and posttest results to demonstrate the improvement in cognitive knowledge on mini-CEX. From September 2010 to August 2011, we implemented a monthly mini-CEX assessment program in our internal medicine residency training. The data of these mini-CEX assessment forms were collected and analyzed.ResultsIn the group of 49 mini-CEX workshop attendees, there was a statistically significant improvement in cognitive knowledge by comparing the pretest and posttest results (67.35 ± 15.25 versus 81.22 ± 10.34, p < 0.001). Among the 863 clinical encounters of mini-CEX, which involved 97 residents and 139 evaluators, 229 (26.5%), 326 (37.8%), and 308 (35.7%) evaluations were completed by the first-year, second-year, and third- year residents separately. We found a statistically significant interaction between level of training and score in dimensions of mini-CEX. The scores in all dimensions measured were better for senior residents. Participation in mini-CEX workshops as a faculty development program strengthened the adherence of trainers to the principles of mini-CEX as a formative assessment in regard to provision of feedback. However, a deficiency in engaging residents' reflection was found.ConclusionsFaculty development is a prerequisite to train evaluators in order to implement a successful mini-CEX assessment program. We demonstrated the effectiveness of our mini-CEX workshops in terms of knowledge acquisition and enhancement of giving feedback when the faculty members used the tool. Further programs on providing effective feedback should be conducted to increase the impact of the mini-CEX as a formative assessment.

Project description:AimIntervention fidelity refers to the degree to which an experimental manipulation has been implemented as intended, but simple, robust methods for quantifying fidelity have not been well documented. Therefore, we aim to illustrate a rigorous quantitative evaluation of intervention fidelity, using data collected during a high-intensity interval training intervention.DesignSingle-group measurement study.MethodsSeventeen adolescents (mean age ± standard deviation [SD] 14.0 ± 0.3 years) attended a 10-week high-intensity interval training intervention, comprising two exercise sessions per week. Sessions consisted of 4-7 45-s maximal effort repetitions, interspersed with 90-s rest. We collected heart rate data at 5-s intervals and recorded the peak heart rate for each repetition. The high-intensity exercise criterion was ≥ 90% of individual maximal heart rate. For each participant, we calculated the proportion of total exercise repetitions exceeding this threshold. A linear mixed model was applied to properly separate the variability in peak heart rate between- and within-subjects. Results are presented both as intention to treat (including missed sessions) and per protocol (only participants with 100% attendance; n=8).ResultsFor intention to treat, the median (interquartile range) proportion of repetitions meeting the high-intensity criterion was 58% (42% to 68%). The mean peak heart rate was 85% of maximal, with a between-subject SD of 7.8 (95% confidence interval 5.4 to 11.3) percentage points and a within-subject SD of 15.1 (14.6 to 15.6) percentage points. For the per protocol analysis, the median proportion of high-intensity repetitions was 68% (47% to 86%). The mean peak heart rate was 91% of maximal, with between- and within-subject SDs of 3.1 (-1.3 to 4.6) and 3.4 (3.2 to 3.6) percentage points, respectively.ConclusionsSynthesising information on exercise session attendance and compliance (exercise intensity) quantifies the intervention dose and informs evaluations of treatment fidelity.

Project description:One of the major debates in implementation research turns around fidelity and adaptation. Fidelity is the degree to which an intervention is implemented as intended by its developers. It is meant to ensure that the intervention maintains its intended effects. Adaptation is the process of implementers or users bringing changes to the original design of an intervention. Depending on the nature of the modifications brought, adaptation could either be potentially positive or could carry the risk of threatening the theoretical basis of the intervention, resulting in a negative effect on expected outcomes. Adaptive interventions are those for which adaptation is allowed or even encouraged. Classical fidelity dimensions and conceptual frameworks do not address the issue of how to adapt an intervention while still maintaining its effectiveness.We support the idea that fidelity and adaptation co-exist and that adaptations can impact either positively or negatively on the intervention's effectiveness. For adaptive interventions, research should answer the question how an adequate fidelity-adaptation balance can be reached. One way to address this issue is by looking systematically at the aspects of an intervention that are being adapted. We conducted fidelity research on the implementation of an empowerment strategy for dengue prevention in Cuba. In view of the adaptive nature of the strategy, we anticipated that the classical fidelity dimensions would be of limited use for assessing adaptations. The typology we used in the assessment-implemented, not-implemented, modified, or added components of the strategy-also had limitations. It did not allow us to answer the question which of the modifications introduced in the strategy contributed to or distracted from outcomes. We confronted our empirical research with existing literature on fidelity, and as a result, considered that the framework for implementation fidelity proposed by Carroll et al. in 2007 could potentially meet our concerns. We propose modifications to the framework to assess both fidelity and adaptation. The modified Carroll et al.'s framework we propose may permit a comprehensive assessment of the implementation fidelity-adaptation balance required when implementing adaptive interventions, but more empirical research is needed to validate it.

Project description:BackgroundDeveloping pragmatic assessment tools to measure clinician use of evidence-based practices is critical to advancing implementation of evidence-based practices in mental health. This case study details our community-partnered process of developing the Therapy Process Observation Coding Scale-Self-Reported Therapist Intervention Fidelity for Youth (TPOCS-SeRTIFY), a pragmatic, clinician-report instrument to measure cognitive behavioral therapy (CBT) delivery.ApproachWe describe a five-step community-partnered development process. Initial goals were to create a self-report instrument that paralleled an existing direct observation measure of clinician delivery of CBT use to facilitate later assessment of measure performance. Cognitive interviews with community clinicians (n = 6) and consultation with CBT experts (n = 6) were used to enhance interpretability and usability as part of an iterative refinement process. The instrument was administered to 247 community clinicians along with an established self-reported measure of clinician delivery of CBT and other treatments to assess preliminary psychometric performance. Preliminary psychometrics were promising.ConclusionOur community-partnered development process showed promising success and can guide future development of pragmatic implementation measures both to facilitate measurement of ongoing implementation efforts and future research aimed at building learning mental health systems.Plain language summaryDeveloping brief, user-friendly, and accurate tools to measure how therapists deliver cognitive behavioral therapy (CBT) in routine practice is important for advancing the reach of CBT into community settings. To date, developing such "pragmatic" measures has been difficult. There is little known about how researchers can best develop these types of assessment tools so that they (1) are easy for clinicians in practice to use and (2) provide valid and useful information about implementation outcomes. As a result, there are few well-validated measures in existence that measure therapist use of CBT that are feasible for use in community practice. This paper contributes to the literature by describing our community-partnered process for developing a measure of therapist use of CBT (Therapy Process Observation Coding Scale -Self-Reported Therapist Intervention Fidelity for Youth; TPOCS-SeRTIFY). This descriptive case study outlines the community-partnered approach we took to develop this measure. This case study will contribute to future research by serving as a guide to others aiming to develop pragmatic implementation measures. In addition, the TPOCS-SeRTIFY is a pragmatic measure of clinician use of CBT that holds promise for its use by both researchers and clinicians to measure the success of CBT implementation efforts.

Project description:Background Implementation of clinical medication reviews in daily practice is scarcely evaluated. The Opti-Med intervention applied a structured approach with external expert teams (pharmacist and physician) to conduct medication reviews. The intervention was effective with respect to resolving drug related problems, but did not improve quality of life. Objective The objective of this process evaluation was to gain more insight into the implementation fidelity of the intervention. Setting Process evaluation alongside a cluster randomized trial in 22 general practices and 518 patients of 65 years and over. Method A mixed methods design using quantitative and qualitative data and the conceptual framework for implementation fidelity was used. Implementation fidelity is defined as the degree to which the various components of an intervention are delivered as intended. Main outcome measure Implementation fidelity for key components of the Opti-Med intervention. Results Patient selection and preparation of the medication analyses were carried out as planned, although mostly by the Opti-Med researchers instead of practice nurses. Medication analyses by expert teams were performed as planned, as well as patient consultations and patient involvement. 48% of the proposed changes in the medication regime were implemented. Cooperation between expert teams members and the use of an online decision-support medication evaluation facilitated implementation. Barriers for implementation were time constraints in daily practice, software difficulties with patient selection and incompleteness of medical files. The degree of embedding of the intervention was found to influence implementation fidelity. The total time investment for healthcare professionals was 94 min per patient. Conclusion Overall, the implementation fidelity was moderate to high for all key components of the Opti-Med intervention. The absence of its effectiveness with respect to quality of life could not be explained by insufficient implementation fidelity.

Project description:Background"We Act" is a health-promoting school intervention comprising an educational, a parental and a school component. The intervention was implemented in 4 Danish public schools with 4 control schools. The objectives were to improve pupils' dietary habits, physical activity, well-being and social capital using the Investigation, Vision, Action & Change (IVAC) health educational methodology. The target group was pupils in grades 5-6. The purpose of this study was to evaluate implementation fidelity and interacting context factors in the intervention schools.MethodsThe Medical Research Council's new guidance for process evaluation was used as a framework. Data were collected concurrently and evenly at the 4 intervention schools through field visits (n?=?43?days), questionnaires (n?=?17 teachers, 52 parents), and interviews (n?=?9 teachers, 4 principals, 52 pupils). The data were analysed separately and via triangulation.ResultsA total of 289 pupils participated, and 22 teachers delivered the educational component in 12 classes. In all schools, the implementation fidelity to the educational methodology was high for the Investigation and Vision phases as the teachers delivered the proposed lessons and activities. However, the implementation fidelity to the Action & Change phase was low, and little change occurred in the schools. The pupils' presentation of their visions did not work as intended as an impact mechanism to prompt actions. The implementation of the parental and the school components was weak. The main context factors influencing implementation fidelity were a poor fit into the school-year plan and weak management support.ConclusionsAlthough 'We Act' was designed to comply with evidence- and theory-based requirements, IVAC and the health-promoting school approach did not result in change. The time dedicated to schools' preparation and competence development may have been too low. This must be considered in future process evaluation research on health-promoting schools and by school health promotion administrators when planning future school interventions.Trial registrationISRCTN85203017.

Project description:BackgroundGiven the urgent need for vaccines and treatments for coronavirus disease 2019 (COVID-19), the Symptoms of Infection with Coronavirus-19 (SIC), a comprehensive, patient-reported outcome (PRO) measure of signs and symptoms associated with COVID-19, was developed in full alignment with current US regulatory guidance to support evaluations of vaccines and treatments in development.MethodsAn initial version of the SIC was developed to address concepts identified through a targeted literature review and consultation with experts in infectious diseases and clinicians routinely managing COVID-19 in a hospital setting. A qualitative study was conducted in sites in the United States among 31 participants aged ≥ 18 years who were English-speaking and willing and able to provide informed consent and a self-reported history by telephone or online method. The measure was refined based on additional feedback from the clinicians and three iterative rounds of combined concept elicitation and cognitive debriefing interviews conducted with patients, caregivers, and healthy volunteers.ResultsAmong 39 scientific articles identified in the literature review, 35 COVID-19 signs and symptoms were reported and confirmed during interviews with clinicians, patients, and caregivers. Patients and healthy participants suggested changes for refining the draft SIC to ensure consistent interpretation and endorsed both the 24-h recall period and use of an 11-point numeric rating scale (NRS) for capturing change in symptom severity. The final version of the SIC captures the daily presence or absence of 30 symptoms and a rating of severity for 25 of the 30 symptoms using an NRS for those symptoms reported as present.ConclusionsThe SIC comprehensively addresses observations described in the literature, by clinicians, and by patients, and captures patients' experiences with COVID-19 in a manner that minimizes complexity and facilitates completion for both patients and healthy volunteers. This measure is thus appropriate for use in clinical trials of both therapeutics and vaccines for COVID-19.

Dataset Information

Development and Testing of a Novel Measure to Assess Fidelity of Implementation: Example of the Mini-AFTERc Intervention.

Background

Objectives

Methods

Results

Conclusion

Clinical trial registration

Publications

Development and Testing of a Novel Measure to Assess Fidelity of Implementation: Example of the Mini-AFTERc Intervention.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets