Dataset Information

Evaluating the Implementation Quality of a Social and Emotional Learning Program: A Mixed Methods Approach.

ABSTRACT: School-based social and emotional learning (SEL) programs have been shown to be effective in producing positive outcomes for adolescents. However, variability in implementation quality can have a negative impact on these program effects. The aim of this current study is to examine the variability in implementation quality for schools implementing the MindOut program and to identify factors that were likely to contribute to this variability. Employing a mixed methods approach, quantitative and qualitative implementation data were collected from teachers (n = 16) and students (n = 280) who participated in the MindOut program. Quantitative indicators were used to score schools' implementation quality across four dimensions (dosage, adherence/fidelity, quality of delivery and participant responsiveness), and these were averaged to determine overall level of implementation (high/low). Qualitative data identified factors that contributed to implementation quality, and factors were then analyzed in accordance with the schools' implementation level grouping. Findings indicated that variability in implementation quality existed both between and within schools. A total of eight schools were assigned as high implementers and another eight as low implementers. Influencing factors were categorized into five themes: (i) program factors, (ii) participant factors, (iii) teacher factors, (iv) school contextual factors, and (v) organizational capacity factors. Several differences between high and low implementers were found in relation to these influencing factors. The findings contribute to the evidence on implementation quality in schools by advancing knowledge on measuring implementation quality across multiple dimensions and informants successfully. These findings can also inform practitioners of the main influencing factors in schools so that strategies can be developed to optimize implementation quality in the future.

SUBMITTER: Dowling K

PROVIDER: S-EPMC7246810 | biostudies-literature | 2020 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluating the Implementation Quality of a Social and Emotional Learning Program: A Mixed Methods Approach.

Dowling Katherine K Barry Margaret M MM

International journal of environmental research and public health 20200507 9

School-based social and emotional learning (SEL) programs have been shown to be effective in producing positive outcomes for adolescents. However, variability in implementation quality can have a negative impact on these program effects. The aim of this current study is to examine the variability in implementation quality for schools implementing the MindOut program and to identify factors that were likely to contribute to this variability. Employing a mixed methods approach, quantitative and qu ...[more]

PMID: 32392698

Similar Datasets

Project description:BACKGROUND:The implementation of evidence-based interventions for borderline personality disorder in community settings is important given that individuals with this diagnosis are often extensive users of both inpatient and outpatient mental health services. Although work in this area is limited, previous studies have identified facilitators and barriers to successful DBT implementation. This study seeks to expand on previous work by evaluating a coordinated implementation of DBT in community settings at a national level. The Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., Implementation Sci. 4:50, 2009) provided structural guidance for this national level coordinated implementation. METHODS:A mixed methods approach was utilised to explore the national multisite implementation of DBT from the perspective of team leaders and therapists who participated in the coordinated training and subsequent implementation of DBT. Qualitative interviews with DBT team leaders (n?=?8) explored their experiences of implementing DBT in their local service and was analysed using content analysis. Quantitative surveys from DBT therapists (n?=?74) examined their experience of multiple aspects of the implementation process including orienting the system, and preparations and support for implementation. Frequencies of responses were calculated. Written qualitative feedback was analysed using content analysis. RESULTS:Five themes were identified from the interview data: team formation, implementation preparation, client selection, service level challenges and team leader role. Participants identified team size and support for the team leader as key points for consideration in DBT implementation. Key challenges encountered were the lack of system support to facilitate phone coaching and a lack of allocated time to focus on DBT. Implementation facilitators included having dedicated team members and support from management. CONCLUSIONS:The barriers and facilitators identified in this study are broadly similar to those reported in previous research. Barriers and facilitators were identified across several domains of the CFIR and are consistent with a recently published DBT implementation Framework (Toms et al., Borderline Personal Disord Emot Dysregul. 6: 2, 2019). Future research should pay particular attention to the domain of characteristics of individuals involved in DBT implementation. The results highlight the importance of a mandated service plan for the coordinated implementation of an evidence-based treatment in a public health service. TRIAL REGISTRATION:ClinicalTrials.gov ID: NCT03180541; Registered June 7th 2017 'retrospectively registered'.

Project description:BACKGROUND:Early supported discharge (ESD) models of care for stroke survivors coordinate inpatient and community services, with the aim of reducing length of stay. While there is an established evidence base around the clinical outcomes of ESD), less is known about the implementation of this approach into existing stroke care service. The aim of this case study was to describe staff perceptions of the implementation of an ESD model of care for stroke survivors at a large metropolitan public hospital in Australia. METHODS:This case study utilised a mixed methods design, which was designed in explicit alignment with the Consolidated Framework for Implementation Research (CFIR). Participants included staff that referred patients for ESD, and staff involved in the planning, implementation or delivery of ESD. Survey data was collected at three time points (ESD commencement, 3 months and 6 months), and focus groups were undertaken at the conclusion of the study. All quantitative data was analysed descriptive, while qualitative data was evaluated using thematic analysis. RESULTS:Results from both sources of data identified changes in staff perceptions of ESD implementation over time. While very few changes were statistically significant, they were diverse patterns of change across the CFIR constructs over time. The characteristics of individuals and ESD characteristics attracted consistently positive perceptions, while patient needs and resources was the most prevalent theme within the data. While perceptions of factors related to the inner setting were mixed, there was a steady improvement in perceptions about the process across the later stages of implementation. CONCLUSIONS:The sophistication of knowledge translation and implementation in modern complex healthcare environments is highlighted by the multiple interactions between the CFIR domains and constructs. While the implementation process described was generally positive and effective, using the CFIR as a framework confirmed that it also entailed some challenges and unanticipated outcomes.

Project description:BACKGROUND:Personal health records (PHRs) provide the opportunity for self-management support, enhancing communication between patients and caregivers, and maintaining and/or improving the quality of chronic disease management. Their implementation is a multi-level and complex process, requiring a holistic approach that takes into account the technology, its users and the context of implementation. The aim of this research is to evaluate the fidelity of a PHR in chronic care (the degree to which it was implemented as intended) in order to explain the found effects. METHODS:A convergent parallel mixed methods design was used, where qualitative and quantitative data were collected in parallel, analyzed separately, and finally merged. Log data of 536 users were used to gain insight into the actual long-term use of the PHR (the dose). Focus group meetings among caregivers (n?=?13) were conducted to assess program differentiation (or intended use). Interviews with caregivers (n?=?28) and usability tests with potential end-users (n?=?13) of the PHR were used to understand the responsiveness and the differences and similarities between the intended and actual use of the PHR. RESULTS:The results of the focus groups showed that services for coaching are strongly associated with monitoring health values and education. However, the PHR was not used that way during the study period. In the interviews, caregivers indicated that they were ignorant on how to deploy the PHR in current working routines. Therefore, they find it difficult to motivate their patients in using the PHR. Participants in the usability study indicate that they would value a PHR in the future, given that the usability will be improved and that the caregivers will use it in daily practice as well. CONCLUSIONS:In this study, actual use of the PHRs by patients was influenced by the responsiveness of caregivers. This responsiveness is likely to be strongly influenced by the perceived support when defining the differentiation and delivery of the PHR. A mixed-methods approach to understand intervention fidelity was of added value in providing explanations for the found effects that could not be revealed by solely focusing on the effectiveness of the technology in an experimental trial.

Project description:BackgroundEffectively scaled-up physical activity interventions are urgently needed to address the high prevalence of physical inactivity. To facilitate scale-up of an efficacious school-based physical activity program (Physical Activity 4 Everyone [PA4E1]), provision of implementation support to physical education (PE) teachers was adapted from face-to-face and paper-based delivery modes to partial delivery via a website. A lack of engagement (usage and subjective experience) with digital delivery modes, including websites, may in part explain the typical reduction in effectiveness of scaled-up interventions that use digital delivery modes. A process evaluation focused on the PA4E1 website was undertaken.ObjectiveThe 2 objectives were to (1) describe the usage of the PA4E1 program website by in-school champions (PE teachers leading the program within their schools) and PE teachers using quantitative methods; (2) examine the usage, subjective experience, and usability of the PA4E1 program website from the perspective of in-school champions using mixed methods.MethodsThe first objective used website usage data collected across all users (n=273) throughout the 9 school terms of the PA4E1 implementation support. The 4 usage measures were sessions, page views, average session duration, and downloads. Descriptive statistics were calculated and explored across the duration of the 26-month program. The second objective used mixed methods, triangulating data from the first objective with data from a think-aloud survey and usability test completed by in-school champions (n=13) at 12 months. Qualitative data were analyzed thematically alongside descriptive statistics from the quantitative data in a triangulation matrix, generating cross-cutting themes using the "following a thread" approach.ResultsFor the first objective, in-school champions averaged 48.0 sessions per user, PE teachers 5.8 sessions. PE teacher sessions were of longer duration (10.5 vs 7.6 minutes) and included more page views (5.4 vs 3.4). The results from the mixed methods analysis for the second objective found 9 themes and 2 meta-themes. The first meta-theme indicated that the website was an acceptable and appropriate delivery mode, and usability of the website was high. The second meta-theme found that the website content was acceptable and appropriate, and identified specific suggestions for improvement.ConclusionsDigital health interventions targeting physical activity often experience issues of lack of user engagement. By contrast, the findings from both the quantitative and mixed methods analyses indicate high usage and overall acceptability and appropriateness of the PA4E1 website to school teachers. The findings support the value of the website within a multidelivery mode implementation intervention to support schools to implement physical activity promoting practices. The analysis identified suggested intervention refinements, which may be adopted for future iterations and further scale-up of the PA4E1 program.Trial registrationAustralian New Zealand Clinical Trials Registry ACTRN12617000681358; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=372870.

Project description:BackgroundThe burden of obesity is high among US veterans, yet many face barriers to engaging in in-person, facility-based treatment programs. To improve access to weight-management services, the Veterans Health Administration (VHA) developed TeleMOVE, a home-based, 82-day curriculum that utilizes in-home messaging devices to promote weight loss in VHA patients facing barriers to accessing facility-based services.ObjectiveThe primary aim was to establish preliminary evidence for the program by comparing outcomes for TeleMOVE with standard, facility-based MOVE weight-management services (group, individual modalities) over the evaluation period based on the number of patients enrolled per site and the program's clinical effectiveness, as demonstrated by average weight lost per patient. The secondary aim was to understand factors influencing TeleMOVE implementation variability across demonstration sites to develop recommendations to improve national program dissemination.MethodsWe employed a formative mixed-methods design to evaluate the phased implementation of TeleMOVE at 9 demonstration sites and compare patient- and site-level measures of program uptake. Data were collected between October 1, 2009 and September 30, 2011. Patient-level program outcomes were extracted from VHA patient care databases to evaluate program enrollment rates and clinical outcomes. To assess preliminary clinical effectiveness, weight loss outcomes for veterans who enrolled in TeleMOVE were compared with outcomes for veterans enrolled in standard MOVE! at each demonstration site, as well as with national averages during the first 2 years of program implementation. For the secondary aim, we invited program stakeholders to participate in 2 rounds of semistructured interviews about aspects of TeleMOVE implementation processes, site-level contextual factors, and program delivery. Twenty-eight stakeholders participated in audio-recorded interviews.ResultsAlthough stakeholders at 3 sites declined to be interviewed, objective program uptake was high at 2 sites, delayed-high at 2 sites, and low at 5 sites. At 6 months post enrollment, the mean weight loss was comparable for TeleMOVE (n=417) and MOVE! (n=1543) participants at -5.2 lb (SD 14.4) and -5.1 lb (SD 12.2), respectively (P=.91). All sites reported high program complexity because TeleMOVE required more staff time per participant than MOVE! due to logistical and technical assistance issues related to the devices. High-uptake sites overcame implementation challenges by leveraging communication networks with stakeholders, adapting the program to patient needs whenever possible, setting programmatic goals and monitoring feedback of results, and taking time to reflect and evaluate on delivery to foster incremental delivery improvements, whereas low-uptake sites reported less leadership support and effective communication among stakeholders.ConclusionsThis implementation evaluation of a clinical telehealth program demonstrated the value of partnership-based research in which researchers not only provided operational leaders with feedback regarding the effectiveness of a new program but also relevant feedback into contextual factors related to program implementation to enable adaptations for national deployment efforts.

Dataset Information

Evaluating the Implementation Quality of a Social and Emotional Learning Program: A Mixed Methods Approach.

Publications

Evaluating the Implementation Quality of a Social and Emotional Learning Program: A Mixed Methods Approach.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets