Dataset Information

Measuring implementation in global mental health: validation of a pragmatic implementation science measure in eastern Ukraine using an experimental vignette design.

ABSTRACT:

Background

There is mounting evidence supporting the effectiveness of task-shifted mental health interventions in low- and middle-income countries (LMIC). However, there has been limited systematic scale-up or sustainability of these programs, indicating a need to study implementation. One barrier to progress is a lack of locally relevant and valid implementation measures. We adapted an existing brief dissemination and implementation (D&I) measure which includes scales for acceptability, appropriateness, feasibility and accessibility for local use and studied its validity and reliability among a sample of consumers in Ukraine.

Methods

Local qualitative data informed adaptation of the measure and development of vignettes to test the reliability and validity. Participants were veterans and internally displaced persons (IDPs) recruited as part of a separate validity study of adapted mental health instruments. We examined internal consistency reliability, test-retest reliability, and construct and criterion validity for each scale on the measure. We randomly assigned half the participants to respond to a vignette depicting existing local psychiatric services which we knew were not well regarded, while the other half was randomized to a vignette describing a potentially more well-implemented mental health service. Criterion validity was assessed by comparing scores on each scale by vignette and by overall summary ratings of the programs described in the vignettes.

Results

N = 169 participated in the qualitative study and N = 153 participated in the validity study. Qualitative findings suggested the addition of several items to the measure and indicated the importance of addressing professionalism/competency of providers in both the scales and the vignettes. Internal consistency reliabilities ranged from α = 0.85 for feasibility to α = 0.91 for appropriateness. Test-rest reliabilities were acceptable to good for all scales (rho: 0.61-0.79). All scales demonstrated substantial and significant differences in average scores by vignette assignment (ORs: 2.21-5.6) and overall ratings (ORs: 5.1-14.47), supporting criterion validity.

Conclusions

This study represents an innovative mixed-methods approach to testing an implementation science measure in contexts outside the United States. Results support the reliability and validity of most scales for consumers in Ukraine. Challenges included large amounts of missing data due to participants' difficulties responding to questions about a hypothetical program.

SUBMITTER: Haroz EE

PROVIDER: S-EPMC6489318 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundValid and reliable measurement of implementation strategies is essential to advancing implementation science; however, this area lags behind the measurement of implementation outcomes and determinants. Clinical supervision is a promising and highly feasible implementation strategy in behavioral healthcare for which pragmatic measures are lacking. This research aimed to develop and psychometrically evaluate a pragmatic measure of clinical supervision conceptualized in terms of two broadly applicable, discrete clinical supervision techniques shown to improve providers' implementation of evidence-based psychosocial interventions-(1) audit and feedback and (2) active learning.MethodsItems were generated based on a systematic review of the literature and administered to a sample of 154 outpatient mental health clinicians serving youth and 181 community-based mental health providers serving adults. Scores were evaluated for evidence of reliability, structural validity, construct-related validity, and measurement invariance across the two samples.ResultsIn sample 1, confirmatory factor analysis (CFA) supported the hypothesized two-factor structure of scores on the Evidence-Based Clinical Supervision Strategies (EBCSS) scale (χ2=5.89, df=4, p=0.208; RMSEA=0.055, CFI=0.988, SRMR=0.033). In sample 2, CFA replicated the EBCSS factor structure and provided discriminant validity evidence relative to an established supervisory alliance measure (χ2=36.12, df=30, p=0.204; RMSEA=0.034; CFI=0.990; SRMR=0.031). Construct-related validity evidence was provided by theoretically concordant associations between EBCSS subscale scores and agency climate for evidence-based practice implementation in sample 1 (d= .47 and .55) as well as measures of the supervision process in sample 2. Multiple group CFA supported the configural, metric, and partial scalar invariance of scores on the EBCSS across the two samples.ConclusionsScores on the EBCSS provide a valid basis for inferences regarding the extent to which behavioral health providers experience audit and feedback and active learning as part of their clinical supervision in both clinic- and community-based behavioral health settings.Trial registrationClinicalTrials.gov NCT04096274 . Registered on 19 September 2019.

Project description:BackgroundA Risk Evaluation and Mitigation Strategy (REMS) is a drug safety program for certain medications with serious safety concerns required by the U.S. Food and Drug Administration (FDA) of manufacturers to implement to help ensure the benefits of the medication outweigh its risks. FDA is encouraging "the research community to develop novel methods for assessing REMS," conveying the unmet need for a standardized evaluation method of these regulatory-mandated healthcare programs. The objective of this research is to evaluate FDA REMS assessment plans using established implementation science frameworks and identify opportunities for strengthening REMS evaluation.MethodsA content analysis was conducted of publicly available assessment plans for all REMS programs (N = 23) approved 1/1/2014-12/31/2018 for new drug applications (NDAs) and biologics license applications (BLAs) requiring FDA-mandated Elements to Assure Safe Use (ETASU). Blinded reviewers critically appraised REMS assessment measures (n = 674) using three established implementation science frameworks: RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance); PRECEDE-PROCEED (Predisposing, Reinforcing, and Enabling Constructs in Educational/Environmental Diagnosis and Evaluation - Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development); and CFIR (Consolidated Framework for Implementation Research). Framework constructs were mapped to REMS Assessment categories as defined by FDA Guidance for Industry to evaluate congruence.ResultsREMS assessment measures demonstrated strong congruence (> 90% mapping rate) with the evaluative constructs of RE-AIM, PRECEDE-PROCEED, and CFIR. Application of the frameworks revealed that REMS assessment measures heavily emphasize implementation and operations, focus less on health outcomes, and do not evaluate program context and design assumptions.ConclusionsImplementation science frameworks have utility for evaluating FDA-mandated drug safety programs including the selection of primary measures to determine whether REMS goals are being met and of secondary measures to evaluate contextual factors affecting REMS effectiveness in varying organizational settings.

Project description:BackgroundFungi play crucial roles in ecosystems and are among the species-richest organism groups on Earth. However, knowledge on their occurrence lags behind the data for animals and plants. Recent analyses of fungal occurrence data from Western, Central and Northern Europe provided important insights into response of fungi to global warming. The consequences of the global changes for biodiversity on a larger geographical scale are not yet understood. Landscapes of Eastern Europe and particularly of eastern Ukraine, with their specific geological history, vegetation and climate, can add substantially new information about fungal diversity in Europe.New informationWe describe the dataset and provide a checklist of aphyllophoroid fungi (non-gilled macroscopic Basidiomycota) from eastern Ukraine sampled in 16 areas between 2007 and 2011. The dataset was managed on the PlutoF biodiversity workbench (http://dx.doi.org/10.15156/BIO/587471) and can also be accessed via Global Biodiversity Information Facility (GBIF, parts of datasets https://doi.org/10.15468/kuspj6 and https://doi.org/10.15468/h7qtfd). This dataset includes 3418 occurences, namely 2727 specimens and 691 observations of fructifications belonging to 349 species of fungi. With these data, the digitised CWU herbarium (V. N. Karazin Kharkiv National University, Ukraine) doubled in size A most detailed description of the substrate's properties and habitat for each record is provided. The specimen records are supplemented by 26 nuclear ribosomal DNA ITS sequences and six 28S sequences. Additionally, 287 photographs depicting diagnostic macro- and microscopic features of fungal fruitbodies as well as studied habitats are linked to the dataset. Most of the specimens have at least one mention in literature and relevant references are displayed as associated with specimen data. In total, 16 publication references are linked to the dataset. The dataset sheds new light on the fungal diversity of Eastern Europe. It is expected to complement other public sources of fungal occurrence information on continental and global levels in addressing macroecological and biogeographical questions.

Project description:BackgroundIn pragmatic trials, on-site partners, rather than researchers, lead intervention delivery, which may result in implementation variation. There is a need to quantitatively measure this variation. Applying the Framework for Implementation Fidelity (FIF), we develop an approach for measuring variability in site-level implementation fidelity. This approach is then applied to measure site-level fidelity in a cluster-randomized pragmatic trial of Music & MemorySM (M&M), a personalized music intervention targeting agitated behaviors in residents living with dementia, in US nursing homes (NHs).MethodsIntervention NHs (N = 27) implemented M&M using a standardized manual, utilizing provided staff trainings and iPods for participating residents. Quantitative implementation data, including iPod metadata (i.e., song title, duration, number of plays), were collected during baseline, 4-month, and 8-month site visits. Three researchers developed four FIF adherence dimension scores. For Details of Content, we independently reviewed the implementation manual and reached consensus on six core M&M components. Coverage was the total number of residents exposed to the music at each NH. Frequency was the percent of participating residents in each NH exposed to M&M at least weekly. Duration was the median minutes of music received per resident day exposed. Data elements were scaled and summed to generate dimension-level NH scores, which were then summed to create a Composite adherence score. NHs were grouped by tercile (low-, medium-, high-fidelity).ResultsThe 27 NHs differed in size, resident composition, and publicly reported quality rating. The Composite score demonstrated significant variation across NHs, ranging from 4.0 to 12.0 [8.0, standard deviation (SD) 2.1]. Scaled dimension scores were significantly correlated with the Composite score. However, dimension scores were not highly correlated with each other; for example, the correlation of the Details of Content score with Coverage was τb = 0.11 (p = 0.59) and with Duration was τb = - 0.05 (p = 0.78). The Composite score correlated with CMS quality star rating and presence of an Alzheimer's unit, suggesting face validity.ConclusionsGuided by the FIF, we developed and used an approach to quantitatively measure overall site-level fidelity in a multi-site pragmatic trial. Future pragmatic trials, particularly in the long-term care environment, may benefit from this approach.Trial registrationClinicaltrials.gov NCT03821844. Registered on 30 January 2019, https://clinicaltrials.gov/ct2/show/NCT03821844 .

Project description:BackgroundEfficacious psychiatric treatments are not consistently deployed in community practice, and clinical outcomes are attenuated compared with those achieved in clinical trials. A major focus for mental health services research is to develop effective and cost-effective strategies that increase the use of evidence-based assessment, prevention, and treatment approaches in community settings.ObjectiveThe goal of this program of research is to apply insights from behavioral economics and participatory design to advance the science and practice of implementing evidence-based practice (EBP) for individuals with psychiatric disorders across the life span.MethodsProject 1 (Assisting Depressed Adults in Primary care Treatment [ADAPT]) is patient-focused and leverages decision-making heuristics to compare ways to incentivize adherence to antidepressant medications in the first 6 weeks of treatment among adults newly diagnosed with depression. Project 2 (App for Strengthening Services In Specialized Therapeutic Support [ASSISTS]) is provider-focused and utilizes normative pressure and social status to increase data collection among community mental health workers treating children with autism. Project 3 (Motivating Outpatient Therapists to Implement: Valuing a Team Effort [MOTIVATE]) explores how participatory design can be used to design organizational-level implementation strategies to increase clinician use of EBPs. The projects are supported by a Methods Core that provides expertise in implementation science, behavioral economics, participatory design, measurement, and associated statistical approaches.ResultsEnrollment for project ADAPT started in 2018; results are expected in 2020. Enrollment for project ASSISTS will begin in 2019; results are expected in 2021. Enrollment for project MOTIVATE started in 2018; results are expected in 2019. Data collection had begun for ADAPT and MOTIVATE when this protocol was submitted.ConclusionsThis research will advance the science of implementation through efforts to improve implementation strategy design, measurement, and statistical methods. First, we will test and refine approaches to collaboratively design implementation strategies with stakeholders (eg, discrete choice experiments and innovation tournaments). Second, we will refine the measurement of mechanisms related to heuristics used in decision making. Third, we will develop new ways to test mechanisms in multilevel implementation trials. This trifecta, coupled with findings from our 3 exploratory projects, will lead to improvements in our knowledge of what causes successful implementation, what variables moderate and mediate the effects of those causal factors, and how best to leverage this knowledge to increase the quality of care for people with psychiatric disorders.Trial registrationClinicalTrials.gov NCT03441399; https://www.clinicaltrials.gov/ct2/show/NCT03441399 (Archived by WebCite at http://www.webcitation.org/74dRbonBD).International registered report identifier (irrid)DERR1-10.2196/12121.