Dataset Information

Using the RE-AIM framework to evaluate internal and external validity of mobile phone-based interventions in diabetes self-management education and support.

ABSTRACT:

Objective

We evaluated the extent to which studies that tested short message service (SMS)- and application (app)-based interventions for diabetes self-management education and support (DSMES) report on factors that inform both internal and external validity as measured by the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework.

Materials and methods

We systematically searched PubMed, Embase, Web of Science, CINAHL (Cumulative Index of Nursing and Allied Health Literature), and IEEE Xplore Digital Library for articles from January 1, 2009, to February 28, 2019. We carried out a multistage screening process followed by email communications with study authors for missing or discrepant information. Two independent coders coded eligible articles using a 23-item validated data extraction tool based on the RE-AIM framework.

Results

Twenty studies (21 articles) were included in the analysis. The comprehensiveness of reporting on the RE-AIM criteria across the SMS- and app-based DSMES studies was low. With respect to internal validity, most interventions were well described and primary clinical or behavioral outcomes were measured and reported. However, gaps exist in areas of attrition, measures of potential negative outcomes, the extent to which the protocol was delivered as intended, and description on delivery agents. Likewise, we found limited information on external validity indicators across adoption, implementation, and maintenance domains.

Conclusions

Reporting gaps were found in internal validity but more so in external validity in the current SMS- and app-based DSMES literature. Because most studies in this review were efficacy studies, the generalizability of these interventions cannot be determined. Future research should adopt the RE-AIM dimensions to improve the quality of reporting and enhance the likelihood of translating research to practice.

SUBMITTER: Yoshida Y

PROVIDER: S-EPMC7309247 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectivesThe purpose of this review was to use RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) to assess the extent to which weight gain prevention studies targeting young adults reported on elements of external validity.DesignSystematic review.Eligibility criteriaArticles of interest included a lifestyle/behavioral intervention targeting weight gain prevention. Eligibility criteria included the following: study design of randomized controlled trials, quasi-randomized control trials, or natural experiments; average participant age between 18 and 35 years; study duration of at least 12 months; and published in English between January 2008 and May 2018. Studies had to report weight or body mass index as a measured outcome and were excluded if they were paired with smoking cessation programs, were conducted in specific groups (ie, pregnant women, breast cancer survivors), or were follow-ups to weight loss studies.Study selectionAfter removing duplicates, the search yielded 11426 articles. Titles and abstracts were screened by 1 reviewer; 144 articles were assessed in a full-text review by 2 reviewers. Discrepancies were resolved by consensus. Nine studies (13 articles) were included in the review.Main outcomes measuresReported elements of the RE-AIM framework.ResultsA total of 9 studies met the selection criteria. All studies lacked full reporting on external validity elements. Of the total of 60 RE-AIM reporting criteria, 8 were reported by all 9 studies, 26 criteria were reported by fewer than 4 studies, and 22 criteria were not reported by any of the studies.DiscussionThere remains inadequate reporting of elements of external validity and generalizability in weight gain prevention studies. This is a significant scientific constraint that limits the information required to disseminate and implement prevention of weight gain interventions for population impact. Standardized reporting may be needed to ensure results that demonstrate not only internal validity but also external validity and generalizability are needed to promote public health impact.

Project description:BackgroundThe monitoring and evaluation of public health programs based on traditional face-to-face interviews in hard-to-reach and unstable regions present many challenges. Mobile phone-based methods are considered to be an effective alternative, but the validity of mobile phone-based data for assessing implementation strength has not been sufficiently studied yet. Nested within an evaluation project for an integrated community case management (iCCM) and family planning program in Mali, this study aimed to assess the validity of a mobile phone-based health provider survey to measure the implementation strength of this program.MethodsFrom July to August 2018, a cross-sectional survey was conducted among the community health workers (ASCs) from six rural districts working with the iCCM and family planning program. ASCs were first reached to complete the mobile phone-based survey; within a week, ASCs were visited in their communities to complete the in-person survey. Both surveys used identical implementation strength tools to collect data on program activities related to iCCM and family planning. Sensitivity and specificity were calculated for each implementation strength indicator collected from the phone-based survey, with the in-person survey as the gold standard. A threshold of ≥ 80% for sensitivity and specificity was considered adequate for evaluation purposes.ResultsOf the 157 ASCs interviewed by mobile phone, 115 (73.2%) were reached in person. Most of the training (2/2 indicators), supervision (2/3), treatment/modern contraceptive supply (9/9), and reporting (3/3) indicators reached the 80% threshold for sensitivity, while only one supervision indicator and one supply indicator reached 80% for specificity. In contrast, most of the stock-out indicators (8/9) reached 80% for specificity, while only two indicators reached the threshold for sensitivity.ConclusionsThe validity of mobile phone-based data was adequate for general training, supervision, and supply indicators for iCCM and family planning. With sufficient mobile phone coverage and reliable mobile network connection, mobile phone-based surveys are useful as an alternative for data collection to assess the implementation strength of general activities in hard-to-reach areas.

Project description:ObjectiveMost formats of currently used community-based health education for cognitive impairment prevention are limited to one-way communication, such as distributing leaflets, pasting posters, or holding a lecture, and they lack comprehensive evaluation. Here we aim to design, test, and evaluate a novel pilot cognitive health education program combined with psychosocial interventions (CHECPI).MethodsWe designed the CHECPI program and tested it among adults aged 60 and over in an aging-friendly community in 2018. Multidimensional cognitive functions were measured by the Montreal Cognitive Assessment (MoCA) before and three months after the CHECPI program. Quantitative and qualitative analyses were performed based on the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework to evaluate the program. Wilcoxon signed-rank tests were used to assess changes in multidimensional cognitive functions.ResultsThe CHECPI program was comprised of 12 courses and introduced 5 kinds of psychosocial interventions. Reach: 28 older adults participated in the program, of whom most were female (n = 22) and younger elderly with an average age of 65.32 years. Effectiveness: 19 participants finished≥6 courses as well as the follow-up survey. Although their MoCA scores did not improve significantly, they had increased their visuospatial ability significantly (with the average score increasing by 0.42). Adoption: the community officers, lecturers, and participants highly recommended the program, but they agreed that the lack of professional instructors may hinder its popularization. Implementation: the program was implemented in full accordance with the pre-program design. Maintenance: three months after the program, 17 participants had maintained at least one of the seven healthy behaviors that were introduced in the program.ConclusionsYounger female elderly were more willing to participate in the program. It enhanced participants' visuospatial ability, but a sufficient number of professional instructors are crucial for large-scale promotion.

Project description:Tobacco cessation after cancer diagnosis leads to better patient outcomes. However, tobacco treatment services are frequently unavailable in cancer care settings, and multilevel implementation challenges can impede uptake of new programs. The National Cancer Institute (NCI) dedicated Cancer Moonshot funding through the Cancer Center Cessation Initiative (C3I) for NCI-Designated Cancer Centers to implement or enhance the implementation of tobacco treatment services. We examined a pragmatic application of the RE-AIM framework (reach, effectiveness, adoption, implementation, and maintenance) to evaluate tobacco treatment programs implemented within Cancer Centers funded through C3I. Using three C3I-funded Centers as examples, we describe how each RE-AIM construct was operationalized to evaluate the implementation of a wide range of cessation services (e.g., tobacco use screening, counseling, Quitline referral, pharmacotherapy) in this heterogeneous group of cancer care settings. We discuss the practical challenges encountered in assessing RE-AIM constructs in real world situations, including using the electronic health record (EHR) to aid in assessment. Reach and effectiveness evaluation required that Centers define the setting(s) where cessation services were implemented (to determine the "denominator"), enumerate the patient population, report current patient tobacco use, patient engagement in tobacco treatment, and 6-month cessation outcomes. To reduce site heterogeneity, increase data accuracy, and reduce burden, reach was frequently captured via standardized EHR enhancements that improved the identification of current smokers and tobacco treatment referrals. Effectiveness was determined by cessation outcomes (30-day point prevalence abstinence at 6-months post-engagement) assessed through a variety of data collection approaches. Adoption was measured by the characteristics and proportion of targeted cancer care settings and clinicians engaged in cessation service delivery. Implementation was assessed by examining the delivery of tobacco screening assessments and intervention components across sites, and provider-level implementation consistency. Maintenance assessments identified whether tobacco treatment services continued in the setting after implementation and documented the sustainability plan and organizational commitment to continued delivery. In sum, this paper demonstrates a pragmatic approach to using RE-AIM as an evaluation framework that yields relevant outcomes on common implementation metrics across widely differing tobacco treatment approaches and settings.

Project description:BackgroundProlonged caregiving of an older adult can cause family caregivers to be overwhelmed, potentially affecting the well-being of both the caregivers and their care-recipients. Carer Matters is a holistic hospital-to-home programme, centred on caregivers' needs as their care-recipients transit from hospital to home. The programme was piloted to support caregivers through caregivers needs assessment, tailored resources, tele-support, training courses, and community support network. This study aimed to examine the feasibility of Carer Matters in a tertiary hospital in Singapore.MethodsThis feasibility study was conducted on the pilot implementation from January to December 2021, during the Covid-19 pandemic. It adopted the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. The study highlighted quantitative data collected from key process indicators, such as number of caregivers screened, assessed on their needs and provided with assistance. Additionally, qualitative data was collected from in-depth interviews with 51 stakeholders involved in the implementation to examine their perspectives and experiences. These included family caregivers, clinician caregiver support nurses, hospital leaders and community partners.ResultsDuring the pilot, 550 caregivers were enrolled. All caregivers received educational resources when they completed the needs assessment, while 69 of them who reported high burden were given tele-support and 252 attended our caregiver training courses. Despite initial recruitment challenges and obstacles to adoption, stakeholders interviewed found Carer Matters to be effective in providing caregivers with emotional support, knowledge and skills that improved their caregiving abilities, and reduced their sense of isolation and caregiving stress. Among caregivers, the training courses were effective with majority of caregivers agreeing that the courses addressed their needs (99%) and improved their knowledge of the relevant disease conditions (97%). Programme maintenance considered among stakeholders included strategies such as multipronged approach in recruiting caregivers and inviting caregiver advocates to share their experiences.ConclusionThis feasibility study highlights that Carer Matters is a valuable component to the ecosystem of support for family caregivers and their care recipients. Carer Matters extends the current patient-centric care model to a more holistic post-discharge continuity of care for both caregivers and their care-recipients, improving and maintaining their overall well-being to better allow transition from hospital-to home.Trial registrationFeasibility Study of Project Carer Matters for Family Caregivers of Persons with Dementia (NCT number: NCT05205135 ).

Project description:Background: The RE-AIM framework has been widely used in health research but it is unclear the extent to which this framework is also used for planning and evaluating health-related programs in clinical and community settings. Our objective was to evaluate how RE-AIM is used in the "real-world" and identify opportunities for improving use outside of research contexts. Methods: We used purposive and snowball sampling to identify clinical and community health programs that used RE-AIM for planning and/or evaluation. Recruitment methods included surveys with email follow-up to funders, implementers, and RE-AIM working group members. We identified 17 programs and conducted structured in-depth interviews with key informants (n = 18). Across RE-AIM dimensions, respondents described motivations, uses, and measures; rated understandability and usefulness; discussed benefits and challenges, strategies to overcome challenges, and resources used. We used descriptive statistics for quantitative ratings, and content analysis for qualitative data. Results: Program content areas included chronic disease management and prevention, healthy aging, mental health, or multiple, often behavioral health-related topics. During planning, most programs considered reach (n = 9), adoption (n = 11), and implementation (n = 12) while effectiveness (n = 7) and maintenance (n = 6) were considered less frequently. In contrast, most programs evaluated all RE-AIM dimensions, ranging from 13 programs assessing maintenance to 15 programs assessing implementation and effectiveness. On five-point scales, all RE-AIM dimensions were rated as easy to understand (Overall M = 4.7 ± 0.5), but obtaining data was rated as somewhat challenging (Overall M = 3.4 ± 0.9). Implementation was the most frequently used dimension to inform program design (M = 4.7 ± 0.6) relative to the other dimensions (3.0-3.9). All dimensions were considered similarly important for decision-making (average M = 4.1 ± 1.4), with the exception of maintenance (M = 3.4 ± 1.7). Qualitative corresponded to the quantitative findings in that RE-AIM was reported to be a practical, easy to understand, and well-established implementation science framework. Challenges included understanding differences among RE-AIM dimensions and data acquisition. Valuable resources included the RE-AIM website and collaborating with an expert. Discussion: RE-AIM is an efficient framework for planning and evaluation of clinical and community-based projects. It provides structure to systematically evaluate health program impact. Programs found planning for and assessing maintenance difficult, providing opportunities for further refinement.

Project description:BackgroundCognitive impairment (CI) is a risk factor for falls due to environmental or living settings, balance, gait and vision impairments, as well as medications. While previous systematic reviews have focused on the effectiveness of fall prevention programs in adults with cognitive impairment, very limited information is available on their implementation. This review examines what aspects of fall prevention interventions for community-dwelling adults with CI have been reported using the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to support successful implementation.MethodsWe examined the included studies from our systematic review, which searched 7 databases for primary and secondary fall prevention interventions involving community-dwelling adults ≥50 years with mild to moderate CI. Reviewers screened citations and extracted data for study characteristics and the 5 dimensions (62 criteria) of the RE-AIM framework.ResultsTwelve randomized or clinical controlled trials (RCTs/CCTs) consisting of 8 exercise interventions, 3 multifactorial interventions, and 1 medication treatment were included in the review. Only 4 of 62 criteria were reported by all 12 included studies and 29 criteria were not reported by any of the studies. Five of the included studies reported on 20 or more of the 62 possible RE-AIM criteria and 3 of these studies self-identified as "feasibility" studies. While Reach was the best-reported construct by the included studies, followed by Effectiveness and Implementation, the criteria within the Adoption and Maintenance constructs were rarely mentioned by these studies. In general, there was also wide variation in how each of the criteria were reported on by study authors.ConclusionBased on the reporting of RE-AIM components in this review, we are unable to make connections to successful intervention components and thus practice-based recommendations for fall prevention in those with CI. The lack of detail regarding implementation approaches greatly limits the interpretation and comparisons across studies to fully inform future research efforts.