Dataset Information

Measuring implementation fidelity in a cluster-randomized pragmatic trial: development and use of a quantitative multi-component approach.

ABSTRACT:

Background

In pragmatic trials, on-site partners, rather than researchers, lead intervention delivery, which may result in implementation variation. There is a need to quantitatively measure this variation. Applying the Framework for Implementation Fidelity (FIF), we develop an approach for measuring variability in site-level implementation fidelity. This approach is then applied to measure site-level fidelity in a cluster-randomized pragmatic trial of Music & Memory^SM (M&M), a personalized music intervention targeting agitated behaviors in residents living with dementia, in US nursing homes (NHs).

Methods

Intervention NHs (N = 27) implemented M&M using a standardized manual, utilizing provided staff trainings and iPods for participating residents. Quantitative implementation data, including iPod metadata (i.e., song title, duration, number of plays), were collected during baseline, 4-month, and 8-month site visits. Three researchers developed four FIF adherence dimension scores. For Details of Content, we independently reviewed the implementation manual and reached consensus on six core M&M components. Coverage was the total number of residents exposed to the music at each NH. Frequency was the percent of participating residents in each NH exposed to M&M at least weekly. Duration was the median minutes of music received per resident day exposed. Data elements were scaled and summed to generate dimension-level NH scores, which were then summed to create a Composite adherence score. NHs were grouped by tercile (low-, medium-, high-fidelity).

Results

The 27 NHs differed in size, resident composition, and publicly reported quality rating. The Composite score demonstrated significant variation across NHs, ranging from 4.0 to 12.0 [8.0, standard deviation (SD) 2.1]. Scaled dimension scores were significantly correlated with the Composite score. However, dimension scores were not highly correlated with each other; for example, the correlation of the Details of Content score with Coverage was τ_b = 0.11 (p = 0.59) and with Duration was τ_b = - 0.05 (p = 0.78). The Composite score correlated with CMS quality star rating and presence of an Alzheimer's unit, suggesting face validity.

Conclusions

Guided by the FIF, we developed and used an approach to quantitatively measure overall site-level fidelity in a multi-site pragmatic trial. Future pragmatic trials, particularly in the long-term care environment, may benefit from this approach.

Trial registration

Clinicaltrials.gov NCT03821844. Registered on 30 January 2019, https://clinicaltrials.gov/ct2/show/NCT03821844 .

SUBMITTER: Olson MB

PROVIDER: S-EPMC8761354 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Implementation research is concerned with bridging the gap between evidence and practice through the study of methods to promote the uptake of research into routine practice. Good quality evidence has been summarised into guideline recommendations to show that peri-operative fasting times could be considerably shorter than patients currently experience. The objective of this trial was to evaluate the effectiveness of three strategies for the implementation of recommendations about peri-operative fasting.A pragmatic cluster randomised trial underpinned by the PARIHS framework was conducted during 2006 to 2009 with a national sample of UK hospitals using time series with mixed methods process evaluation and cost analysis. Hospitals were randomised to one of three interventions: standard dissemination (SD) of a guideline package, SD plus a web-based resource championed by an opinion leader, and SD plus plan-do-study-act (PDSA). The primary outcome was duration of fluid fast prior to induction of anaesthesia. Secondary outcomes included duration of food fast, patients' experiences, and stakeholders' experiences of implementation, including influences. ANOVA was used to test differences over time and interventions.Nineteen acute NHS hospitals participated. Across timepoints, 3,505 duration of fasting observations were recorded. No significant effect of the interventions was observed for either fluid or food fasting times. The effect size was 0.33 for the web-based intervention compared to SD alone for the change in fluid fasting and was 0.12 for PDSA compared to SD alone. The process evaluation showed different types of impact, including changes to practices, policies, and attitudes. A rich picture of the implementation challenges emerged, including inter-professional tensions and a lack of clarity for decision-making authority and responsibility.This was a large, complex study and one of the first national randomised controlled trials conducted within acute care in implementation research. The evidence base for fasting practice was accepted by those participating in this study and the messages from it simple; however, implementation and practical challenges influenced the interventions' impact. A set of conditions for implementation emerges from the findings of this study, which are presented as theoretically transferable propositions that have international relevance.ISRCTN18046709--Peri-operative Implementation Study Evaluation (POISE).

Project description:BACKGROUND:Access to high-quality antenatal care services has been shown to be beneficial for maternal and child health. In 2016, the WHO published evidence-based recommendations for antenatal care that aim to improve utilization, quality of care, and the patient experience. Prior research in Nepal has shown that a lack of social support, birth planning, and resources are barriers to accessing services in rural communities. The success of CenteringPregnancy and participatory action women's groups suggests that group care models may both improve access to care and the quality of care delivered through women's empowerment and the creation of social networks. We present a group antenatal care model in rural Nepal, designed and implemented by the healthcare delivery organization Nyaya Health Nepal, as well as an assessment of implementation outcomes. METHODS:The study was conducted at Bayalata Hospital in Achham, Nepal, via a public private partnership between the Nepali non-profit, Nyaya Health Nepal, and the Ministry of Health and Population, with financial and technical assistance from the American non-profit, Possible. We implemented group antenatal care as a prospective non-randomized cluster-controlled, type I hybrid effectiveness-implementation study in six village clusters. The implementation approach allows for iterative improvement in design, making changes to improve the quality of the intervention. Assessments of implementation process and model fidelity were undertaken using a mobile checklist completed by nurse supervisors, and observation forms completed by program leadership. We evaluated data quarterly using descriptive statistics to identify trends. Qualitative interviews and team communications were analyzed through immersion crystallization to identify major themes that evolved during the implementation process. RESULTS:A total of 141 group antenatal sessions were run during the study period. This paper reports on implementation results, whereas we analyze and present patient-level effectiveness outcomes in a complementary paper in this journal. There was high process fidelity to the model, with 85.7% (95% CI 77.1-91.5%) of visits completing all process elements, and high content fidelity, with all village clusters meeting the minimum target frequency for 80% of topics. The annual per capita cost for group antenatal care was 0.50 USD. Qualitative analysis revealed the compromise of stable gestation-matched composition of the group members in order to make the intervention feasible. Major adaptations were made in training, documentation, feedback and logistics. CONCLUSION:Group antenatal care provided in collaboration with local government clinics has the potential to provide accessible and high quality antenatal care to women in rural Nepal. The intervention is a feasible and affordable alternative to individual antenatal care. Our experience has shown that adaptation from prior models was important for the program to be successful in the local context within the national healthcare system. TRIAL REGISTRATION:ClinicalTrials.gov Identifier: NCT02330887, registered 01/05/2015, retroactively registered.

Project description:BackgroundChild-oriented goal-setting in pediatric rehabilitation may improve child motivation, engagement in therapy, child outcomes related to therapy, and service delivery efficiency. The primary objective of this trial is to determine the effectiveness of a principles-driven, child-focused approach to goal-setting, Enhancing Child Engagement in Goal-Setting (ENGAGE), on pediatric rehabilitation outcomes compared to usual practice. The three secondary objectives are to 1) compare costs and secondary outcomes of the ENGAGE approach to usual practice, 2) determine the influence of child, parent and therapist characteristics on child engagement in therapy and rehabilitation outcomes, and 3) identify barriers and facilitators to the implementation of ENGAGE.MethodsThis research protocol describes a pragmatic, multi-site, cluster, effectiveness-implementation (hybrid type 1 design) randomized controlled trial. Therapists (n = 12 clusters of two therapists) at participating sites (n = 6) will be randomized to 1) the ENGAGE intervention group, or 2) usual care (control) using a computer-generated, permuted-block randomization sequence with site as a stratification variable designed by a statistician (RR). Each therapist will recruit four children 5-12 years old with neurodevelopmental conditions (n = 96), who will receive ENGAGE or usual care, according to therapist group allocation. ENGAGE therapists will be trained to use a 'toolbox' of evidence-driven, theory-informed principles to optimize child and parent motivation, engagement in the goal-setting process, and performance feedback strategies. Outcomes include goal performance (primary outcome), engagement in therapy, functional abilities, participation, and parent and child quality of life. Qualitative interviews with children, parents, ENGAGE therapists, and managers will explore challenges to implementation and potential mitigation strategies. Mixed effects multiple linear regression models will be developed for each outcome to assess group differences adjusted for clustering. A cost-effectiveness analysis will combine cost and a measure of effectiveness into an incremental cost-effectiveness ratio. Qualitative data on implementation will be analyzed inductively (thematic analysis) and deductively using established implementation science frameworks.DiscussionThis study will evaluate the effects of collaborative goal-setting in pediatric rehabilitation and inform effective implementation of child-focused goal-setting practices.Trial registrationNCT05017363 (registered August 23, 2021 on ClinicalTrials.gov).

Project description:BackgroundUntil now, there is no clear overview of how fidelity is assessed in school-based obesity prevention programmes. In order to move the field of obesity prevention programmes forward, the current review aimed to 1) identify which fidelity components have been measured in school-based obesity prevention programmes; 2) identify how fidelity components have been measured; and 3) score the quality of these methods.MethodsStudies published between January 2001-October 2017 were selected from searches in PubMed, EMBASE, PsycINFO, CINAHL, Cochrane Library and ERIC. We included studies examining the fidelity of obesity prevention programmes (nutrition and/or physical activity and/or sitting) at school (children aged 4-18 year) measuring at least one component of implementation fidelity. A data extraction was performed to identify which and how fidelity components were measured. Thereafter, a quality assessment was performed to score the quality of these methods. We scored each fidelity component on 7 quality criteria. Each fidelity component was rated high (>?75% positive), moderate (50-75%) or low (<?50%).ResultsOf the 26,294 retrieved articles, 73 articles reporting on 63 different studies were included in this review. In 17 studies a process evaluation was based on a theoretical framework. In total, 120 fidelity components were measured across studies: dose was measured most often (N =?50), followed by responsiveness (N =?36), adherence (N =?26) and quality of delivery (N =?8). There was substantial variability in how fidelity components were defined as well as how they were measured. Most common methods were observations, logbooks and questionnaires targeting teachers. The quality assessment scores ranged from 0 to 86%; most fidelity components scored low quality (n =?77).ConclusionsThere is no consensus on the operationalisation of concepts and methods used for assessing fidelity in school-based obesity prevention programmes and the quality of methods used is weak. As a result, we call for more consensus on the concepts and clear reporting on the methods employed for measurements of fidelity to increase the quality of fidelity measurements. Moreover, researchers should focus on the relation between fidelity and programme outcomes and determine to what extent adaptations to programmes have been made, whilst still being effective.

Project description:BackgroundMost people with opioid use disorder (OUD) never receive treatment. Medication treatment of OUD in primary care is recommended as an approach to increase access to care. The PRimary Care Opioid Use Disorders treatment (PROUD) trial tests whether implementation of a collaborative care model (Massachusetts Model) using a nurse care manager (NCM) to support medication treatment of OUD in primary care increases OUD treatment and improves outcomes. Specifically, it tests whether implementation of collaborative care, compared to usual primary care, increases the number of days of medication for OUD (implementation objective) and reduces acute health care utilization (effectiveness objective). The protocol for the PROUD trial is presented here.MethodsPROUD is a hybrid type III cluster-randomized implementation trial in six health care systems. The intervention consists of three implementation strategies: salary for a full-time NCM, training and technical assistance for the NCM, and requiring that three primary care providers have DEA waivers to prescribe buprenorphine. Within each health system, two primary care clinics are randomized: one to the intervention and one to Usual Primary Care. The sample includes all patients age 16-90 who visited the randomized primary care clinics from 3 years before to 2 years after randomization (anticipated to be > 170,000). Quantitative data are derived from existing health system administrative data, electronic medical records, and/or health insurance claims ("electronic health records," [EHRs]). Anonymous staff surveys, stakeholder debriefs, and observations from site visits, trainings and technical assistance provide qualitative data to assess barriers and facilitators to implementation. The outcome for the implementation objective (primary outcome) is a clinic-level measure of the number of patient days of medication treatment of OUD over the 2 years post-randomization. The patient-level outcome for the effectiveness objective (secondary outcome) is days of acute care utilization [e.g. urgent care, emergency department (ED) and/or hospitalizations] over 2 years post-randomization among patients with documented OUD prior to randomization.DiscussionThe PROUD trial provides information for clinical leaders and policy makers regarding potential benefits for patients and health systems of a collaborative care model for management of OUD in primary care, tested in real-world diverse primary care settings. Trial registration # NCT03407638 (February 28, 2018); CTN-0074 https://clinicaltrials.gov/ct2/show/NCT03407638?term=CTN-0074&draw=2&rank=1.

Project description:BackgroundThere is mounting evidence supporting the effectiveness of task-shifted mental health interventions in low- and middle-income countries (LMIC). However, there has been limited systematic scale-up or sustainability of these programs, indicating a need to study implementation. One barrier to progress is a lack of locally relevant and valid implementation measures. We adapted an existing brief dissemination and implementation (D&I) measure which includes scales for acceptability, appropriateness, feasibility and accessibility for local use and studied its validity and reliability among a sample of consumers in Ukraine.MethodsLocal qualitative data informed adaptation of the measure and development of vignettes to test the reliability and validity. Participants were veterans and internally displaced persons (IDPs) recruited as part of a separate validity study of adapted mental health instruments. We examined internal consistency reliability, test-retest reliability, and construct and criterion validity for each scale on the measure. We randomly assigned half the participants to respond to a vignette depicting existing local psychiatric services which we knew were not well regarded, while the other half was randomized to a vignette describing a potentially more well-implemented mental health service. Criterion validity was assessed by comparing scores on each scale by vignette and by overall summary ratings of the programs described in the vignettes.ResultsN = 169 participated in the qualitative study and N = 153 participated in the validity study. Qualitative findings suggested the addition of several items to the measure and indicated the importance of addressing professionalism/competency of providers in both the scales and the vignettes. Internal consistency reliabilities ranged from α = 0.85 for feasibility to α = 0.91 for appropriateness. Test-rest reliabilities were acceptable to good for all scales (rho: 0.61-0.79). All scales demonstrated substantial and significant differences in average scores by vignette assignment (ORs: 2.21-5.6) and overall ratings (ORs: 5.1-14.47), supporting criterion validity.ConclusionsThis study represents an innovative mixed-methods approach to testing an implementation science measure in contexts outside the United States. Results support the reliability and validity of most scales for consumers in Ukraine. Challenges included large amounts of missing data due to participants' difficulties responding to questions about a hypothetical program.

Project description:BACKGROUND:The Pragmatic Trial of Video Education in Nursing Homes (PROVEN) is one of the first large pragmatic randomized clinical trials (pRCTs) to be conducted in U.S. nursing homes (N = 119 intervention and N = 241 control across two health-care systems). The trial aims to evaluate the effectiveness of a suite of videos to improve advance care planning (ACP) for nursing home patients. This report uses mixed methods to explore the optimal and suboptimal conditions necessary for implementation fidelity within pRCTs in nursing homes. METHODS:PROVEN's protocol required designated facility champions to offer an ACP video to long-stay patients every 6 months during the 18-month implementation period. Champions completed a video status report, stored within electronic medical records, each time a video was offered. Data from the report were used to derive each facility's adherence rate (i.e., cumulative video offer). Qualitative interviews held after 15 months with champions were purposively sampled from facilities within the highest and lowest adherence rates (i.e., those in the top and bottom quintiles). Two researchers analyzed interview data thematically using a deductive approach based upon six domains of the revised Conceptual Framework for Implementation Fidelity (CFIF). Matrices were developed to compare coded narratives by domain across facility adherence status. RESULTS:In total, 28 interviews involving 33 champions were analyzed. Different patterns were observed across high- versus low-adherence facilities for five CFIF domains. In low-adherence nursing homes, (1) there were limited implementation resources (Context), (2) there was often a perceived negative patient or family responsiveness to the program (Participant Responsiveness), and (3) champions were reticent in offering the videos (Recruitment). In high-adherence nursing homes, (1) there was more perceived patient and family willingness to engage in the program (Participant Responsiveness), (2) champions supplemented the video with ACP conversations (Quality of Delivery), (3) there were strategic approaches to recruitment (Recruitment), and (4) champions appreciated external facilitation (Strategies to Facilitate Implementation). CONCLUSIONS:Critical lessons for implementing pRCTs in nursing homes emerged from this report: (1) flexible fidelity is important (i.e., delivering core elements of an intervention while permitting the adaptation of non-core elements), (2) reciprocal facilitation is vital (i.e., early and ongoing stakeholder engagement in research design and, reciprocally, researchers' and organizational leaders' ongoing support of the implementation), and (3) organizational and champion readiness should be formally assessed early and throughout implementation to facilitate remediation. TRIAL REGISTRATION:ClinicalTrials.gov, NCT02612688. Registered on 19 November 2015.