Dataset Information

Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity.

ABSTRACT:

Objectives

The aim of this study was to (1) demonstrate the development and testing of tools and procedures designed to monitor and assess the integrity of a complex intervention for chronic pain (COping with persistent Pain, Effectiveness Research into Self-management (COPERS) course); and (2) make recommendations based on our experiences.

Design

Fidelity assessment of a two-arm randomised controlled trial intervention, assessing the adherence and competence of the facilitators delivering the intervention.

Setting

The intervention was delivered in the community in two centres in the UK: one inner city and one a mix of rural and urban locations.

Participants

403 people with chronic musculoskeletal pain were enrolled in the intervention arm and 300 attended the self-management course. Thirty lay and healthcare professionals were trained and 24 delivered the courses (2 per course). We ran 31 courses for up to 16 people per course and all were audio recorded.

Interventions

The course was run over three and a half days; facilitators delivered a semistructured manualised course.

Outcomes

We designed three measures to evaluate fidelity assessing adherence to the manual, competence and overall impression.

Results

We evaluated a random sample of four components from each course (n=122). The evaluation forms were reliable and had good face validity. There were high levels of adherence in the delivery: overall adherence was two (maximum 2, IQR 1.67-2.00), facilitator competence exhibited more variability, and overall competence was 1.5 (maximum 2, IQR 1.25-2.00). Overall impression was three (maximum 4, IQR 2.00-3.00).

Conclusions

Monitoring and assessing adherence and competence at the point of intervention delivery can be realised most efficiently by embedding the principles of fidelity measurement within the design stage of complex interventions and the training and assessment of those delivering the intervention. More work is necessary to ensure that more robust systems of fidelity evaluation accompany the growth of complex interventions.

Trial registration isrctn no

ISRCTN24426731.

SUBMITTER: Mars T

PROVIDER: S-EPMC3831105 | biostudies-literature | 2013 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity.

Mars Tom T Ellard David D Carnes Dawn D Homer Kate K Underwood Martin M Taylor Stephanie J C SJ

BMJ open 20131115 11

<h4>Objectives</h4>The aim of this study was to (1) demonstrate the development and testing of tools and procedures designed to monitor and assess the integrity of a complex intervention for chronic pain (COping with persistent Pain, Effectiveness Research into Self-management (COPERS) course); and (2) make recommendations based on our experiences.<h4>Design</h4>Fidelity assessment of a two-arm randomised controlled trial intervention, assessing the adherence and competence of the facilitators d ...[more]

PMID: 24240140

Similar Datasets

Project description:BackgroundResearchers face many decisions in developing a measurement tool and protocol for monitoring fidelity to complex interventions. The current study uses data evaluating a nutrition education intervention, Together, We Inspire Smart Eating (WISE), in a preschool setting to explore issues of source, timing, and frequency of fidelity monitoring.MethodsThe overall study from which these data are drawn was a pre/post design with an implementation-focused process evaluation. Between 2013 and 2016, researchers monitored fidelity to evidence-based components of the WISE intervention in 49 classrooms in two Southern states. Data collectors obtained direct assessment of fidelity on a monthly basis in study classrooms. Research staff requested that educators provide indirect assessment on a weekly basis. We used mean comparisons (t-tests), correlations (Pearson's r), and scatterplots to compare the direct and indirect assessments.ResultsNo mean comparisons were statistically different. Correlations of direct and indirect assessments of the same component for the same month ranged between - 0.51 (p = 0.01) and 0.54 (p = 0.001). Scatterplots illustrate that negative correlations can be driven by individuals who are over reporting (i.e., self-report bias) and that near zero correlations approximate the ideal situation (i.e., both raters identify high fidelity).ConclusionOur findings illustrate that, on average, observed and self-reports may seem consistent despite weak correlations and individual cases of extreme over reporting by those implementing the intervention. The nature of the component to which fidelity is being monitored as well as the timing within the context of the intervention are important factors to consider when selecting the type of assessment and frequency of fidelity monitoring.Trial registrationNCT03075085 Registered 20 February 2017. Trial registration corresponds to the funding that supported the writing of this manuscript, not the data collection. The original study was not a trial and was collected without registration. However, the data reported here provided foundational preliminary data for the trial.

Project description:In this paper, we introduce the Intervention Mapping (IM) taxonomy of behaviour change methods and its potential to be developed into a coding taxonomy. That is, although IM and its taxonomy of behaviour change methods are not in fact new, because IM was originally developed as a tool for intervention development, this potential was not immediately apparent. Second, in explaining the IM taxonomy and defining the relevant constructs, we call attention to the existence of parameters for effectiveness of methods, and explicate the related distinction between theory-based methods and practical applications and the probability that poor translation of methods may lead to erroneous conclusions as to method-effectiveness. Third, we recommend a minimal set of intervention characteristics that may be reported when intervention descriptions and evaluations are published. Specifying these characteristics can greatly enhance the quality of our meta-analyses and other literature syntheses. In conclusion, the dynamics of behaviour change are such that any taxonomy of methods of behaviour change needs to acknowledge the importance of, and provide instruments for dealing with, three conditions for effectiveness for behaviour change methods. For a behaviour change method to be effective: (1) it must target a determinant that predicts behaviour; (2) it must be able to change that determinant; (3) it must be translated into a practical application in a way that preserves the parameters for effectiveness and fits with the target population, culture, and context. Thus, taxonomies of methods of behaviour change must distinguish the specific determinants that are targeted, practical, specific applications, and the theory-based methods they embody. In addition, taxonomies should acknowledge that the lists of behaviour change methods will be used by, and should be used by, intervention developers. Ideally, the taxonomy should be readily usable for this goal; but alternatively, it should be clear how the information in the taxonomy can be used in practice. The IM taxonomy satisfies these requirements, and it would be beneficial if other taxonomies would be extended to also meet these needs.

Project description:Behaviour change interventions are likely to be reproducible only if reported clearly. We assessed whether the behaviour change technique taxonomy version 1 (BCTTv1), with and without training in identifying BCTs, improves the clarity and replicability of written reports of observed behaviour change interventions.Three studies assessed effects of using and training in the use of BCTTv1 on the clarity and replicability of intervention descriptions written after observing videos of smoking cessation interventions. Study 1 examined the effects of using and not using BCTTv1. Study 2 examined the effects of using BCTTv1 and training in use of BCTTv1 compared no use and no training. Study 3 employed a within-group design to assess change in descriptions written before and after training. One-hundred and 66 'writers' watched videos of behaviour change interventions and wrote descriptions of the active components delivered. In all studies, the participants' written descriptions were evaluated by (i) 12 'raters' (untrained in BCTTv1) for clarity and replicability and (ii) 12 'coders' (trained in BCTTv1) for reliability of BCT coding. Writers rated the usability and accessibility of using BCTTv1 to write descriptions.Ratings of clarity and replicability did not differ between groups in study 1 (all ps > 0.05), were poorer for trained users in study 2 (all ps < 0.01) and improved following training in study 3 (all ps < 0.05). BCT identification was more reliable from descriptions written by trained BCTTv1 users (p < 0.05; study 2) but not simple use of BCTTv1 (p = 0.93; study 1) or by writers who had written a description without BCTTv1, before training (p = 0.50; study 3). Writers reported that using BCTTv1 was difficult but 'useful', 'good' and 'desirable' and that their descriptions would be clear and replicable (all means above mid-point of the scale).Effects of training to use BCTTv1 on the quality of written reports of observed interventions were mixed, with some suggestion of improved clarity and replicability of reporting in the within- (study 3) but not the between-group studies (studies 1 and 2). Potential benefits of using BCTTv1 may have been limited by the artificial nature and time constraints of the task.

Project description:PurposeUnderstanding the effectiveness of complex, face-to-face health behaviour change interventions requires high-quality measures to assess fidelity of delivery and engagement. This systematic review aimed to (1) identify the types of measures used to monitor fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions and (2) describe the reporting of psychometric and implementation qualities.MethodsElectronic databases were searched, systematic reviews and reference lists were hand-searched, and 21 experts were contacted to identify articles. Studies that quantitatively measured fidelity of delivery of, and/or engagement with, a complex, face-to-face health behaviour change intervention for adults were included. Data on interventions, measures, and psychometric and implementation qualities were extracted and synthesized using narrative analysis.ResultsSixty-six studies were included: 24 measured both fidelity of delivery and engagement, 20 measured fidelity of delivery, and 22 measured engagement. Measures of fidelity of delivery included observation (n = 17; 38.6%), self-report (n = 15; 34%), quantitatively rated qualitative interviews (n = 1; 2.3%), or multiple measures (n = 11; 25%). Measures of engagement included self-report (n = 18; 39.1%), intervention records (n = 11; 24%), or multiple measures (n = 17; 37%). Fifty-one studies (77%) reported at least one psychometric or implementation quality; 49 studies (74.2%) reported at least one psychometric quality, and 17 studies (25.8%) reported at least one implementation quality.ConclusionFewer than half of the reviewed studies measured both fidelity of delivery of, and engagement with complex, face-to-face health behaviour change interventions. More studies reported psychometric qualities than implementation qualities. Interpretation of intervention outcomes from fidelity of delivery and engagement measurements may be limited due to a lack of reporting of psychometric and implementation qualities. Statement of contribution What is already known on this subject? Evidence of fidelity and engagement is needed to understand effectiveness of complex interventions Evidence of fidelity and engagement are rarely reported High-quality measures are needed to measure fidelity and engagement What does this study add? Evidence that indicators of quality of measures are reported in some studies Evidence that psychometric qualities are reported more frequently than implementation qualities A recommendation for intervention evaluations to report indicators of quality of fidelity and engagement measures.

Project description:BackgroundIntervention fidelity is the degree to which interventions have been implemented as intended by their developers. Assessing fidelity is crucial for accurate interpretation of intervention effectiveness, but fidelity is often poorly addressed within trials of complex healthcare interventions. The reasons for this are unclear, and information on the use of methods to enhance and assess fidelity in trials of complex interventions remains insufficient. This study aimed to explore the knowledge, practice and attitudes towards intervention fidelity amongst researchers, triallists and healthcare professionals involved with the design and conduct of trials of complex healthcare interventions.MethodsAn online survey consisting of closed and open-ended questions exploring four sections (Demographics, Fidelity knowledge, Practice and Attitudes) was conducted. This was an opportunistic sample of individuals with experience of direct involvement in trials of complex healthcare interventions (e.g. design/development, conduct, evaluation).ResultsData from 264 participants representing 15 countries were analysed. The majority (65.9%, n = 174) of participants identified themselves as 'Researchers'. The majority of participants were familiar with the term "intervention fidelity" (69.7%, n = 184) and indicated that fidelity is important (89.7%, n = 236). Mean self-reported understanding of fidelity was moderate. Although 68% (n = 182) had previously used strategies to assess (e.g. audio/video-recording sessions) and enhance (e.g. training manual) fidelity in trials of complex interventions, only a limited proportion of participants indicated always reporting these strategies in subsequent publications (30.9%, n = 56). Poor knowledge or understanding was the most commonly cited barrier to addressing intervention fidelity in trials (77.4%, n = 202). Over half of respondents (52.1%, n = 137) had never completed specific fidelity training or research, and the vast majority (89.7%, n = 236) would welcome specific training in this area.ConclusionDespite good awareness of intervention fidelity and its importance, poor knowledge and understanding appears to be a key factor limiting how intervention fidelity is addressed in trials of complex interventions. Participants identified a need for further training and education in this area. Additionally, clarification of the terminology, definition and components of intervention fidelity would facilitate better understanding of the concept. A discrepancy between participants' use of fidelity strategies and subsequent reporting raises concerns around inadequate fidelity reporting in the trials literature.

Project description:Background: Contextual factors such as an intervention's setting are key to understanding how interventions to change behaviour have their effects and patterns of generalisation across contexts. The intervention's setting is not consistently reported in published reports of evaluations. Using ontologies to specify and classify intervention setting characteristics enables clear and reproducible reporting, thus aiding replication, implementation and evidence synthesis. This paper reports the development of a Setting Ontology for behaviour change interventions as part of a Behaviour Change Intervention Ontology, currently being developed in the Wellcome Trust funded Human Behaviour-Change Project. Methods: The Intervention Setting Ontology was developed following methods for ontology development used in the Human Behaviour-Change Project: 1) Defining the ontology's scope, 2) Identifying key entities by reviewing existing classification systems (top-down) and 100 published behaviour change intervention reports (bottom-up), 3) Refining the preliminary ontology by literature annotation of 100 reports, 4) Stakeholder reviewing by 23 behavioural science and public health experts to refine the ontology, 5) Assessing inter-rater reliability of using the ontology by two annotators familiar with the ontology and two annotators unfamiliar with it, 6) Specifying ontological relationships between setting entities and 7) Making the Intervention Setting Ontology machine-readable using Web Ontology Language (OWL) and publishing online. Re sults: The Intervention Setting Ontology consists of 72 entities structured hierarchically with two upper-level classes: Physical setting including Geographic location, Attribute of location (including Area social and economic condition, Population and resource density sub-levels) and Intervention site (including Facility, Transportation and Outdoor environment sub-levels), as well as Social setting. Inter-rater reliability was found to be 0.73 (good) for those familiar with the ontology and 0.61 (acceptable) for those unfamiliar with it. Conclusion: The Intervention Setting Ontology can be used to code information from diverse sources, annotate the setting characteristics of existing intervention evaluation reports and guide future reporting.

Project description:The structural integrity of the corticospinal tract (CST) after stroke is closely linked to the degree of motor impairment. Simple and reliable methods of assessing white matter integrity within the CST would facilitate the use of this measure in routine clinical practice. Commonly, diffusion tensor imaging is used to measure voxel-wise fractional anisotropy (FA) in a variety of regions of interest (ROIs) representing the CST. Several methods are currently in use with no consensus about which approach is best. ROIs are usually either the whole CST or the posterior limb of the internal capsule (PLIC). These are created manually on brain images or with reference to an individual's CST determined by tractography. Once the ROI has been defined, the FA can be reported as an absolute measure from the ipsilesional side or as a ratio in comparison to the contralesional side. Both corticospinal tracking and manual ROI definition in individual stroke patients are time consuming and subject to bias. Here, we investigated whether using a CST template derived from healthy volunteers was a feasible method for defining the appropriate ROI within which to measure changes in FA. We reconstructed the CST connecting the primary motor cortex to the ipsilateral pons in 23 age-matched control subjects and 21 stroke patients. An average healthy CST template was created from the 23 control subjects. For each patient, FA values were then calculated for both the template CST and for their own CST. We compared patients' FA metrics between the two tracts by considering four measures (FA in the ipsilesional side, FA in the contralesional side, FA ratio of the ipsilesional side to the contralesional side and FA asymmetry between the two sides) and in two tract-based ROIs (whole tract and tract section traversing the PLIC). There were no significant differences in FA metrics for either method, except for contralesional FA. Furthermore, we found that FA metrics relating to CST damage all correlated with motor ability post-stroke equally well. These results suggest that the healthy CST template could be a surrogate structure for defining tract-based ROIs with which to measure stroke patients' FA metrics, avoiding the necessity for CST tracking in individual patients. CST template-based automated quantification of structural integrity would greatly facilitate implementation of practical clinical applications of diffusion tensor imaging.

Dataset Information

Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity.

Objectives

Design

Setting

Participants

Interventions

Outcomes

Results

Conclusions

Trial registration isrctn no

Publications

Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets