Dataset Information

Fidelity is not easy! Challenges and guidelines for assessing fidelity in complex interventions.

ABSTRACT:

Background

Fidelity in complex behavioural interventions is underexplored and few comprehensive or detailed fidelity studies report on specific procedures for monitoring fidelity. Using Bellg's popular Treatment Fidelity model, this paper aims to increase understanding of how to practically and comprehensively assess fidelity in complex, group-level, interventions.

Approach and lessons learned

Drawing on our experience using a mixed methods approach to assess fidelity in the INFORM study (Improving Nursing home care through Feedback On perfoRMance data-INFORM), we report on challenges and adaptations experienced with our fidelity assessment approach and lessons learned. Six fidelity assessment challenges were identified: (1) the need to develop succinct tools to measure fidelity given tools tend to be intervention specific, (2) determining which components of fidelity (delivery, receipt, enactment) to emphasize, (3) unit of analysis considerations in group-level interventions, (4) missing data problems, (5) how to respond to and treat fidelity 'failures' and 'deviations' and lack of an overall fidelity assessment scheme, and (6) ensuring fidelity assessment doesn't threaten internal validity.

Recommendations and conclusions

Six guidelines, primarily applicable to group-level studies of complex interventions, are described to help address conceptual, methodological, and practical challenges with fidelity assessment in pragmatic trials. The current study offers guidance to researchers regarding key practical, methodological, and conceptual challenges associated with assessing fidelity in pragmatic trials. Greater attention to fidelity assessment and publication of fidelity results through detailed studies such as this one is critical for improving the quality of fidelity studies and, ultimately, the utility of published trials.

Trial registration

ClinicalTrials.gov NCT02695836. Registered on February 24, 2016.

SUBMITTER: Ginsburg LR

PROVIDER: S-EPMC8164256 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:OBJECTIVES:To understand whether interventions are effective, we need to know whether the interventions are delivered as planned (with fidelity) and engaged with. To measure fidelity and engagement effectively, high-quality measures are needed. We outline a five-step method which can be used to develop quality measures of fidelity and engagement for complex health interventions. We provide examples from a fidelity study conducted within an evaluation of an intervention aimed to increase independence in dementia. METHODS:We propose five steps that can be systematically used to develop fidelity checklists for researchers, providers, and participants to measure fidelity and engagement. These steps include the following: (1) reviewing previous measures, (2) analysing intervention components and developing a framework outlining the content of the intervention, (3) developing fidelity checklists and coding guidelines, (4) obtaining feedback about the content and wording of checklists and guidelines, and (5) piloting and refining checklists and coding guidelines to assess and improve reliability. RESULTS:Three fidelity checklists that can be used reliably were developed to measure fidelity of and engagement with, the Promoting Independence in Dementia (PRIDE) intervention. As these measures were designed to be used by researchers, providers, and participants, we developed two versions of the checklists: one for participants and one for researchers and providers. CONCLUSIONS:The five steps that we propose can be used to develop psychometrically robust and implementable measures of fidelity and engagement for complex health interventions that can be used by different target audiences. By considering quality when developing measures, we can be more confident in the interpretation of intervention outcomes drawn from fidelity and engagement studies. Statement of contribution What is already known on the subject? Fidelity and engagement can be measured using a range of methods, such as observation and self-report. Studies seldom report psychometric and implementation qualities of fidelity measures. What does this study add? A method for developing fidelity and engagement measures for complex health interventions. Guidance on how to consider quality when developing fidelity and engagement measures.

Project description:BackgroundResearchers face many decisions in developing a measurement tool and protocol for monitoring fidelity to complex interventions. The current study uses data evaluating a nutrition education intervention, Together, We Inspire Smart Eating (WISE), in a preschool setting to explore issues of source, timing, and frequency of fidelity monitoring.MethodsThe overall study from which these data are drawn was a pre/post design with an implementation-focused process evaluation. Between 2013 and 2016, researchers monitored fidelity to evidence-based components of the WISE intervention in 49 classrooms in two Southern states. Data collectors obtained direct assessment of fidelity on a monthly basis in study classrooms. Research staff requested that educators provide indirect assessment on a weekly basis. We used mean comparisons (t-tests), correlations (Pearson's r), and scatterplots to compare the direct and indirect assessments.ResultsNo mean comparisons were statistically different. Correlations of direct and indirect assessments of the same component for the same month ranged between - 0.51 (p = 0.01) and 0.54 (p = 0.001). Scatterplots illustrate that negative correlations can be driven by individuals who are over reporting (i.e., self-report bias) and that near zero correlations approximate the ideal situation (i.e., both raters identify high fidelity).ConclusionOur findings illustrate that, on average, observed and self-reports may seem consistent despite weak correlations and individual cases of extreme over reporting by those implementing the intervention. The nature of the component to which fidelity is being monitored as well as the timing within the context of the intervention are important factors to consider when selecting the type of assessment and frequency of fidelity monitoring.Trial registrationNCT03075085 Registered 20 February 2017. Trial registration corresponds to the funding that supported the writing of this manuscript, not the data collection. The original study was not a trial and was collected without registration. However, the data reported here provided foundational preliminary data for the trial.

Project description:ObjectivesThe aim of this study was to (1) demonstrate the development and testing of tools and procedures designed to monitor and assess the integrity of a complex intervention for chronic pain (COping with persistent Pain, Effectiveness Research into Self-management (COPERS) course); and (2) make recommendations based on our experiences.DesignFidelity assessment of a two-arm randomised controlled trial intervention, assessing the adherence and competence of the facilitators delivering the intervention.SettingThe intervention was delivered in the community in two centres in the UK: one inner city and one a mix of rural and urban locations.Participants403 people with chronic musculoskeletal pain were enrolled in the intervention arm and 300 attended the self-management course. Thirty lay and healthcare professionals were trained and 24 delivered the courses (2 per course). We ran 31 courses for up to 16 people per course and all were audio recorded.InterventionsThe course was run over three and a half days; facilitators delivered a semistructured manualised course.OutcomesWe designed three measures to evaluate fidelity assessing adherence to the manual, competence and overall impression.ResultsWe evaluated a random sample of four components from each course (n=122). The evaluation forms were reliable and had good face validity. There were high levels of adherence in the delivery: overall adherence was two (maximum 2, IQR 1.67-2.00), facilitator competence exhibited more variability, and overall competence was 1.5 (maximum 2, IQR 1.25-2.00). Overall impression was three (maximum 4, IQR 2.00-3.00).ConclusionsMonitoring and assessing adherence and competence at the point of intervention delivery can be realised most efficiently by embedding the principles of fidelity measurement within the design stage of complex interventions and the training and assessment of those delivering the intervention. More work is necessary to ensure that more robust systems of fidelity evaluation accompany the growth of complex interventions.Trial registration isrctn noISRCTN24426731.

Project description:BackgroundIntervention fidelity is the degree to which interventions have been implemented as intended by their developers. Assessing fidelity is crucial for accurate interpretation of intervention effectiveness, but fidelity is often poorly addressed within trials of complex healthcare interventions. The reasons for this are unclear, and information on the use of methods to enhance and assess fidelity in trials of complex interventions remains insufficient. This study aimed to explore the knowledge, practice and attitudes towards intervention fidelity amongst researchers, triallists and healthcare professionals involved with the design and conduct of trials of complex healthcare interventions.MethodsAn online survey consisting of closed and open-ended questions exploring four sections (Demographics, Fidelity knowledge, Practice and Attitudes) was conducted. This was an opportunistic sample of individuals with experience of direct involvement in trials of complex healthcare interventions (e.g. design/development, conduct, evaluation).ResultsData from 264 participants representing 15 countries were analysed. The majority (65.9%, n?=?174) of participants identified themselves as 'Researchers'. The majority of participants were familiar with the term "intervention fidelity" (69.7%, n?=?184) and indicated that fidelity is important (89.7%, n?=?236). Mean self-reported understanding of fidelity was moderate. Although 68% (n?=?182) had previously used strategies to assess (e.g. audio/video-recording sessions) and enhance (e.g. training manual) fidelity in trials of complex interventions, only a limited proportion of participants indicated always reporting these strategies in subsequent publications (30.9%, n?=?56). Poor knowledge or understanding was the most commonly cited barrier to addressing intervention fidelity in trials (77.4%, n?=?202). Over half of respondents (52.1%, n?=?137) had never completed specific fidelity training or research, and the vast majority (89.7%, n?=?236) would welcome specific training in this area.ConclusionDespite good awareness of intervention fidelity and its importance, poor knowledge and understanding appears to be a key factor limiting how intervention fidelity is addressed in trials of complex interventions. Participants identified a need for further training and education in this area. Additionally, clarification of the terminology, definition and components of intervention fidelity would facilitate better understanding of the concept. A discrepancy between participants' use of fidelity strategies and subsequent reporting raises concerns around inadequate fidelity reporting in the trials literature.

Project description:BackgroundControlled-feeding trials are challenging to design and administer in a free-living setting. There is a need to share methods and best practices for diet design, delivery, and standard adherence metrics.ObjectivesThis report describes menu planning, implementing, and monitoring of controlled diets for an 8-wk free-living trial comparing a diet pattern based on the Dietary Guidelines for Americans (DGA) and a more typical American diet (TAD) pattern based on NHANES 2009-2010. The objectives were to 1) provide meals that were acceptable, portable, and simple to assemble at home; 2) blind the intervention diets to the greatest extent possible; and 3) use tools measuring adherence to determine the success of the planned and implemented menu.MethodsMenus were blinded by placing similar dishes on the 2 intervention diets but changing recipes. Adherence was monitored using daily food checklists, a real-time dashboard of scores from daily checklists, weigh-backs of containers returned, and 24-h urinary nitrogen recoveries. Proximate analyses of diet composites were used to compare the macronutrient composition of the composite and planned menu.ResultsMeeting nutrient intake recommendations while scaling menus for individual energy intake amounts and food portions was most challenging for vitamins D and E, the sodium-to-potassium ratio, dietary fiber, and fatty acid composition. Dietary adherence for provided foods was >95%, with no differences between groups. Urinary nitrogen recoveries were ∼80% relative to nitrogen intake and not different between groups. Composite proximate analysis matched the plan for dietary fat, protein, and carbohydrates. Dietary fiber was ∼2.5 g higher in the TAD composite compared with the planned menu, but ∼7.4 g lower than the DGA composite.ConclusionsBoth DGA and TAD diets were acceptable to most participants. This conclusion was supported by self-reported consumption, quantitative weigh-backs of provided food, and urinary nitrogen recovery. Dietary adherence measures in controlled-feeding trials would benefit from standard protocols to promote uniformity across studies. The trial is registered at clinicaltrials.gov as NCT02298725.

Project description:Public deliberation, or deliberative democracy, is a method used to elicit informed perspectives and justifiable solutions to ethically fraught or contentious issues that affect multiple stakeholder groups with conflicting interests. Deliberative events bring together stakeholders (deliberants) who are provided with empirical evidence on the central issue or concern and then asked to discuss the evidence, consider the issue from a societal perspective, and collectively work toward a justifiable resolution. There is increasing interest in this method, which warrants clear guidance for evaluating the quality of its use in research. Most of the existing literature on measuring deliberation quality emphasizes the quality of deliberants' inputs (eg, engagement and evidence of compromise) during deliberative sessions. Fewer researchers have framed quality in terms of facilitator inputs, and these researchers tend to examine inputs that are consistent with generic group processes. The theory, process, and purpose of public deliberation, however, are distinct from those of focus groups or other group-based discussions and warrant a mechanism for measuring quality in terms of facilitator fidelity to the principles and processes of deliberative democracy. In our public deliberation on ethical conflicts in minor consent for biomedical HIV prevention research, we assessed facilitator fidelity to these principles and processes because we believe that such assessments serve as a component of a comprehensive evaluation of overall deliberation quality. We examined verbatim facilitator remarks in the deliberation transcripts and determined whether they aligned with the 6 principles of public deliberation: equal participation, respect for the opinions of others, adoption of a societal perspective, reasoned justification of ideas, expression of diverse opinions, and compromise or movement toward consensus. In this tutorial, we describe the development of a blueprint to guide researchers in assessing facilitator fidelity, share 3 templates that will assist them in the task, and describe the results of our assessment of facilitator fidelity in 1 of the 4 sites in which we conducted deliberations.