Dataset Information

Developing quality fidelity and engagement measures for complex health interventions.

ABSTRACT: OBJECTIVES:To understand whether interventions are effective, we need to know whether the interventions are delivered as planned (with fidelity) and engaged with. To measure fidelity and engagement effectively, high-quality measures are needed. We outline a five-step method which can be used to develop quality measures of fidelity and engagement for complex health interventions. We provide examples from a fidelity study conducted within an evaluation of an intervention aimed to increase independence in dementia. METHODS:We propose five steps that can be systematically used to develop fidelity checklists for researchers, providers, and participants to measure fidelity and engagement. These steps include the following: (1) reviewing previous measures, (2) analysing intervention components and developing a framework outlining the content of the intervention, (3) developing fidelity checklists and coding guidelines, (4) obtaining feedback about the content and wording of checklists and guidelines, and (5) piloting and refining checklists and coding guidelines to assess and improve reliability. RESULTS:Three fidelity checklists that can be used reliably were developed to measure fidelity of and engagement with, the Promoting Independence in Dementia (PRIDE) intervention. As these measures were designed to be used by researchers, providers, and participants, we developed two versions of the checklists: one for participants and one for researchers and providers. CONCLUSIONS:The five steps that we propose can be used to develop psychometrically robust and implementable measures of fidelity and engagement for complex health interventions that can be used by different target audiences. By considering quality when developing measures, we can be more confident in the interpretation of intervention outcomes drawn from fidelity and engagement studies. Statement of contribution What is already known on the subject? Fidelity and engagement can be measured using a range of methods, such as observation and self-report. Studies seldom report psychometric and implementation qualities of fidelity measures. What does this study add? A method for developing fidelity and engagement measures for complex health interventions. Guidance on how to consider quality when developing fidelity and engagement measures.

SUBMITTER: Walton H

PROVIDER: S-EPMC7004004 | biostudies-literature | 2020 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Developing quality fidelity and engagement measures for complex health interventions.

Walton Holly H Spector Aimee A Williamson Morgan M Tombor Ildiko I Michie Susan S

British journal of health psychology 20191106 1

<h4>Objectives</h4>To understand whether interventions are effective, we need to know whether the interventions are delivered as planned (with fidelity) and engaged with. To measure fidelity and engagement effectively, high-quality measures are needed. We outline a five-step method which can be used to develop quality measures of fidelity and engagement for complex health interventions. We provide examples from a fidelity study conducted within an evaluation of an intervention aimed to increase ...[more]

PMID: 31693797

Similar Datasets

Project description:PurposeUnderstanding the effectiveness of complex, face-to-face health behaviour change interventions requires high-quality measures to assess fidelity of delivery and engagement. This systematic review aimed to (1) identify the types of measures used to monitor fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions and (2) describe the reporting of psychometric and implementation qualities.MethodsElectronic databases were searched, systematic reviews and reference lists were hand-searched, and 21 experts were contacted to identify articles. Studies that quantitatively measured fidelity of delivery of, and/or engagement with, a complex, face-to-face health behaviour change intervention for adults were included. Data on interventions, measures, and psychometric and implementation qualities were extracted and synthesized using narrative analysis.ResultsSixty-six studies were included: 24 measured both fidelity of delivery and engagement, 20 measured fidelity of delivery, and 22 measured engagement. Measures of fidelity of delivery included observation (n = 17; 38.6%), self-report (n = 15; 34%), quantitatively rated qualitative interviews (n = 1; 2.3%), or multiple measures (n = 11; 25%). Measures of engagement included self-report (n = 18; 39.1%), intervention records (n = 11; 24%), or multiple measures (n = 17; 37%). Fifty-one studies (77%) reported at least one psychometric or implementation quality; 49 studies (74.2%) reported at least one psychometric quality, and 17 studies (25.8%) reported at least one implementation quality.ConclusionFewer than half of the reviewed studies measured both fidelity of delivery of, and engagement with complex, face-to-face health behaviour change interventions. More studies reported psychometric qualities than implementation qualities. Interpretation of intervention outcomes from fidelity of delivery and engagement measurements may be limited due to a lack of reporting of psychometric and implementation qualities. Statement of contribution What is already known on this subject? Evidence of fidelity and engagement is needed to understand effectiveness of complex interventions Evidence of fidelity and engagement are rarely reported High-quality measures are needed to measure fidelity and engagement What does this study add? Evidence that indicators of quality of measures are reported in some studies Evidence that psychometric qualities are reported more frequently than implementation qualities A recommendation for intervention evaluations to report indicators of quality of fidelity and engagement measures.

Project description:BackgroundIt is important to evaluate fidelity of delivery and engagement during feasibility trials. However, there is little guidance on how to systematically develop strategies to improve implementation if problems arise. We aimed to: 1) Assess fidelity of delivery and engagement, 2) Identify factors influencing fidelity of delivery and engagement, and 3) Develop strategies to improve fidelity of delivery of, and engagement with, a complex intervention to improve independence in dementia, within a feasibility trial.MethodsA mixed methods evaluation of an intervention that aimed to improve independence in dementia. To assess fidelity of delivery and engagement, observation and self-report methods were used: 60% of audio-recorded intervention sessions were transcribed and reliably rated for fidelity. Providers (n =?12) and people with dementia/supporters (n =?34) were asked to complete checklists after each session. Descriptive statistics were used to analyse the data. To identify factors influencing fidelity and engagement, one-to-one semi-structured interviews were conducted with providers (n =?8), people with dementia (n =?7) and supporters (n =?7). Thematic analysis and content analysis were used to analyse data. To develop strategies, we followed four steps proposed by the authors of the Behaviour Change Wheel (1. Understanding the behaviour, 2. Identifying intervention functions, 3. Specifying intervention content, 4. Identifying mode of delivery).ResultsResearcher ratings indicated moderate fidelity and provider/participant ratings indicated high fidelity of delivery. Knowledge, providers' attributes, ease of adaptation of the intervention in relation to participants' needs and logistical considerations influenced fidelity. We developed four strategies to improve fidelity of delivery of PRIDE: 1) showing a video, 2) giving an instruction sheet, 3) giving time to practice and 4) providing continued support. Participants reported high levels of engagement. Participants' attributes, capability and opportunity influenced engagement. We developed four strategies to improve engagement with PRIDE: 1) a session summary document, 2) clear instructions, 3) time to practice activity and 4) providing regular compulsory telephone support.ConclusionFidelity of delivery and engagement are complex behaviours. This manuscript provides an example of how the Behaviour Change Wheel can be used during a feasibility trial to systematically develop strategies to improve implementation of complex interventions.

Project description:BackgroundTuberculosis (TB) management can be challenging in low- and middle-income countries (LMICs) not only because of its high burden but also the prolonged treatment period involving multiple drugs. With rapid development in mobile technology, mobile health (mHealth) interventions or using a mobile device for TB management has gained popularity. Despite the potential usefulness of mHealth interventions for TB, few studies have quantitatively synthesized evidence on its effectiveness, presumably because of variability in outcome measures reported in the literature.ObjectiveThe aim of this systematic review was to evaluate the outcome measures reported in TB mHealth literature in LMICs.MethodsMEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews were searched to identify mHealth intervention studies for TB (published up to May 2018) that reported any type of outcome measures. The extracted information included the study setting, types of mHealth technology used, target population, study design, and categories of outcome measures. Outcomes were classified into 13 categories including treatment outcome, adherence, process measure, perception, technical outcome, and so on. The qualitative synthesis of evidence focused on the categories of outcome measures reported by the type of mHealth interventions.ResultsA total of 27 studies were included for the qualitative synthesis of evidence. The study designs varied widely, ranging from randomized controlled trials to economic evaluations. A total of 12 studies adopted short message service (SMS), whereas 5 studies used SMS in combination with additional technologies or mobile apps. The study populations were also diverse, including patients with TB, patients with TB/HIV, health care workers, and general patients attending a clinic. There was a wide range of variations in the definition of outcome measures across the studies. Among the diverse categories of outcome measures, treatment outcomes have been reported in 14 studies, but only 6 of them measured the outcome according to the standard TB treatment definitions by the World Health Organization.ConclusionsThis critical evaluation of outcomes reported in mHealth studies for TB management suggests that substantial variability exists in reporting outcome measures. To overcome the challenges in evidence synthesis for mHealth interventions, this study can provide insights into the development of a core set of outcome measures by intervention type and study design.

Project description:Background:Researchers face many decisions in developing a measurement tool and protocol for monitoring fidelity to complex interventions. The current study uses data evaluating a nutrition education intervention, Together, We Inspire Smart Eating (WISE), in a preschool setting to explore issues of source, timing, and frequency of fidelity monitoring. Methods:The overall study from which these data are drawn was a pre/post design with an implementation-focused process evaluation. Between 2013 and 2016, researchers monitored fidelity to evidence-based components of the WISE intervention in 49 classrooms in two Southern states. Data collectors obtained direct assessment of fidelity on a monthly basis in study classrooms. Research staff requested that educators provide indirect assessment on a weekly basis. We used mean comparisons (t-tests), correlations (Pearson's r), and scatterplots to compare the direct and indirect assessments. Results:No mean comparisons were statistically different. Correlations of direct and indirect assessments of the same component for the same month ranged between -?0.51 (p?=?0.01) and 0.54 (p?=?0.001). Scatterplots illustrate that negative correlations can be driven by individuals who are over reporting (i.e., self-report bias) and that near zero correlations approximate the ideal situation (i.e., both raters identify high fidelity). Conclusion:Our findings illustrate that, on average, observed and self-reports may seem consistent despite weak correlations and individual cases of extreme over reporting by those implementing the intervention. The nature of the component to which fidelity is being monitored as well as the timing within the context of the intervention are important factors to consider when selecting the type of assessment and frequency of fidelity monitoring. Trial registration:NCT03075085 Registered 20 February 2017. Trial registration corresponds to the funding that supported the writing of this manuscript, not the data collection. The original study was not a trial and was collected without registration. However, the data reported here provided foundational preliminary data for the trial.

Project description:One of the major debates in implementation research turns around fidelity and adaptation. Fidelity is the degree to which an intervention is implemented as intended by its developers. It is meant to ensure that the intervention maintains its intended effects. Adaptation is the process of implementers or users bringing changes to the original design of an intervention. Depending on the nature of the modifications brought, adaptation could either be potentially positive or could carry the risk of threatening the theoretical basis of the intervention, resulting in a negative effect on expected outcomes. Adaptive interventions are those for which adaptation is allowed or even encouraged. Classical fidelity dimensions and conceptual frameworks do not address the issue of how to adapt an intervention while still maintaining its effectiveness.We support the idea that fidelity and adaptation co-exist and that adaptations can impact either positively or negatively on the intervention's effectiveness. For adaptive interventions, research should answer the question how an adequate fidelity-adaptation balance can be reached. One way to address this issue is by looking systematically at the aspects of an intervention that are being adapted. We conducted fidelity research on the implementation of an empowerment strategy for dengue prevention in Cuba. In view of the adaptive nature of the strategy, we anticipated that the classical fidelity dimensions would be of limited use for assessing adaptations. The typology we used in the assessment-implemented, not-implemented, modified, or added components of the strategy-also had limitations. It did not allow us to answer the question which of the modifications introduced in the strategy contributed to or distracted from outcomes. We confronted our empirical research with existing literature on fidelity, and as a result, considered that the framework for implementation fidelity proposed by Carroll et al. in 2007 could potentially meet our concerns. We propose modifications to the framework to assess both fidelity and adaptation. The modified Carroll et al.'s framework we propose may permit a comprehensive assessment of the implementation fidelity-adaptation balance required when implementing adaptive interventions, but more empirical research is needed to validate it.

Project description:BackgroundVideo is a versatile and popular medium for digital health interventions. As mobile device and app technology advances, it is likely that video-based interventions will become increasingly common. Although clinic waiting rooms are complex and busy environments, they offer the opportunity to facilitate engagement with video-based digital interventions as patients wait to see their providers. However, to increase efficiency in public health, leverage the scalability and low cost of implementing digital interventions, and keep up with rapidly advancing technology and user needs, more design and development guidance is needed for video-based tailored interventions.ObjectiveWe provide a tutorial for digital intervention researchers and developers to efficiently design and develop video-based tailored digital health interventions. We describe the challenges and solutions encountered with Positive Health Check (PHC), a hybrid app used to deliver a brief, interactive, individually tailored video-based HIV behavioral counseling intervention. PHC uses video clips and multimedia digital assets to deliver intervention content, including interactive tailored messages and graphics, a repurposed animated video, and patient and provider handouts generated in real time by PHC.MethodsWe chronicle multiple challenges and solutions for the following: (1) using video as a medium to enhance user engagement, (2) navigating the complexity of linking a database of video clips with other digital assets, and (3) identifying the main steps involved in building an app that will seamlessly deliver to users individually tailored messages, graphics, and handouts.ResultsWe leveraged video to enhance user engagement by featuring "video doctors," full-screen video, storyboards, and streamlined scripts. We developed an approach to link the database of video clips with other digital assets through script coding and flow diagrams of algorithms to deliver a tailored user experience. We identified the steps to app development by using keyframes to design the integration of video and digital assets, using agile development methods to gather iterative feedback from multidisciplinary teams, and creating an intelligent data-driven back-end solution to tailor message delivery to individual users.ConclusionsVideo-based digital health interventions will continue to play an important role in the future of HIV prevention and treatment, as well as other clinical health practices. However, facilitating the adoption of an HIV video intervention in HIV clinical settings is a work in progress. Our experience in designing and developing PHC presented unique challenges due to the extensive use of a large database of videos tailored individually to each user. Although PHC focuses on promoting the health and well-being of persons with HIV, the challenges and solutions presented in this tutorial are transferable to the design and development of video-based digital health interventions focused on other areas of health.

Project description:Objectives: The aim of this study was to (1) demonstrate the development and testing of tools and procedures designed to monitor and assess the integrity of a complex intervention for chronic pain (COping with persistent Pain, Effectiveness Research into Self-management (COPERS) course); and (2) make recommendations based on our experiences.Design: Fidelity assessment of a two-arm randomised controlled trial intervention, assessing the adherence and competence of the facilitators delivering the intervention.Setting: The intervention was delivered in the community in two centres in the UK: one inner city and one a mix of rural and urban locations.Participants: 403 people with chronic musculoskeletal pain were enrolled in the intervention arm and 300 attended the self-management course. Thirty lay and healthcare professionals were trained and 24 delivered the courses (2 per course). We ran 31 courses for up to 16 people per course and all were audio recorded.Interventions: The course was run over three and a half days; facilitators delivered a semistructured manualised course.Outcomes: We designed three measures to evaluate fidelity assessing adherence to the manual, competence and overall impression.Results: We evaluated a random sample of four components from each course (n=122). The evaluation forms were reliable and had good face validity. There were high levels of adherence in the delivery: overall adherence was two (maximum 2, IQR 1.67-2.00), facilitator competence exhibited more variability, and overall competence was 1.5 (maximum 2, IQR 1.25-2.00). Overall impression was three (maximum 4, IQR 2.00-3.00).Conclusions: Monitoring and assessing adherence and competence at the point of intervention delivery can be realised most efficiently by embedding the principles of fidelity measurement within the design stage of complex interventions and the training and assessment of those delivering the intervention. More work is necessary to ensure that more robust systems of fidelity evaluation accompany the growth of complex interventions.Trial registration isrctn no: ISRCTN24426731.

Dataset Information

Developing quality fidelity and engagement measures for complex health interventions.

Publications

Developing quality fidelity and engagement measures for complex health interventions.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets