Dataset Information

Use of a Machine Learning Program to Correctly Triage Incoming Text Messaging Replies From a Cardiovascular Text-Based Secondary Prevention Program: Feasibility Study.

ABSTRACT: BACKGROUND:SMS text messaging programs are increasingly being used for secondary prevention, and have been shown to be effective in a number of health conditions including cardiovascular disease. SMS text messaging programs have the potential to increase the reach of an intervention, at a reduced cost, to larger numbers of people who may not access traditional programs. However, patients regularly reply to the SMS text messages, leading to additional staffing requirements to monitor and moderate the patients' SMS text messaging replies. This additional staff requirement directly impacts the cost-effectiveness and scalability of SMS text messaging interventions. OBJECTIVE:This study aimed to test the feasibility and accuracy of developing a machine learning (ML) program to triage SMS text messaging replies (ie, identify which SMS text messaging replies require a health professional review). METHODS:SMS text messaging replies received from 2 clinical trials were manually coded (1) into "Is staff review required?" (binary response of yes/no); and then (2) into 12 general categories. Five ML models (Naïve Bayes, OneVsRest, Random Forest Decision Trees, Gradient Boosted Trees, and Multilayer Perceptron) and an ensemble model were tested. For each model run, data were randomly allocated into training set (2183/3118, 70.01%) and test set (935/3118, 29.98%). Accuracy for the yes/no classification was calculated using area under the receiver operating characteristics curve (AUC), false positives, and false negatives. Accuracy for classification into 12 categories was compared using multiclass classification evaluators. RESULTS:A manual review of 3118 SMS text messaging replies showed that 22.00% (686/3118) required staff review. For determining need for staff review, the Multilayer Perceptron model had highest accuracy (AUC 0.86; 4.85% false negatives; and 4.63% false positives); with addition of heuristics (specified keywords) fewer false negatives were identified (3.19%), with small increase in false positives (7.66%) and AUC 0.79. Application of this model would result in 26.7% of SMS text messaging replies requiring review (true + false positives). The ensemble model produced the lowest false negatives (1.43%) at the expense of higher false positives (16.19%). OneVsRest was the most accurate (72.3%) for the 12-category classification. CONCLUSIONS:The ML program has high sensitivity for identifying the SMS text messaging replies requiring staff input; however, future research is required to validate the models against larger data sets. Incorporation of an ML program to review SMS text messaging replies could significantly reduce staff workload, as staff would not have to review all incoming SMS text messages. This could lead to substantial improvements in cost-effectiveness, scalability, and capacity of SMS text messaging-based interventions.

SUBMITTER: Lowres N

PROVIDER: S-EPMC7327598 | biostudies-literature | 2020 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Use of a Machine Learning Program to Correctly Triage Incoming Text Messaging Replies From a Cardiovascular Text-Based Secondary Prevention Program: Feasibility Study.

Lowres Nicole N Duckworth Andrew A Redfern Julie J Thiagalingam Aravinda A Chow Clara K CK

JMIR mHealth and uHealth 20200616 6

<h4>Background</h4>SMS text messaging programs are increasingly being used for secondary prevention, and have been shown to be effective in a number of health conditions including cardiovascular disease. SMS text messaging programs have the potential to increase the reach of an intervention, at a reduced cost, to larger numbers of people who may not access traditional programs. However, patients regularly reply to the SMS text messages, leading to additional staffing requirements to monitor and ...[more]

PMID: 32543439

Similar Datasets

Project description:BackgroundDespite promising data in Western countries, there is a dearth of research into the efficacy of text messaging-based smoking cessation programs in other settings, including the Middle East, where smoking prevalence rates are higher.ObjectiveThis paper reports cessation rates observed in SMS Turkey, a text messaging-based smoking cessation program for adult smokers in Ankara, Turkey.MethodsThis study was a small-scale, parallel-group randomized controlled trial (RCT) conducted in Ankara, Turkey. Participants were adult daily smokers who were seriously thinking about quitting in the next 15 days and living in Ankara, Turkey. The text messaging intervention, SMS Turkey, provided 6 weeks of daily messages aimed at giving participants skills to help them quit smoking. Messages were sent in an automated fashion, except 2 days and 7 days after the initial quit day. On days 2 and 7, the research assistant manually assigned participants to content "paths" based on whether they were still not smoking or had relapsed. The control arm received a brochure that provided similar information about smoking cessation. The main outcome measure was self-reported 3-month sustained abstinence, verified by carbon monoxide (CO) readings. Neither participants nor researchers were blinded to arm assignment.ResultsThe 151 participants were randomly assigned to 1 of 2 groups: 76 to the SMS Turkey intervention group and 75 to the brochure control group. Using intention to treat, all 151 participants were included in analyses. Three-month cessation trends were not significantly higher in the intervention group: 11% intervention vs 5% control had quit (χ(2)(1)=1.4, P=.24; R(2)=2.0, 95% CI 0.62-6.3). When the sample was stratified by sex, female intervention participants (14%, n=5) were significantly more likely to have quit at 3 months than female control participants (0%, n=0; χ(2)(1)=3.7, P=.05). Among light smokers (ie, those smoking less than 20 cigarettes per day), intervention participants (17%, n=5) also were significantly more likely to have quit compared to control participants (0%, n=0; χ(2)(1)=5.3, P=.02). We noted no difference in cessation rates for males or heavy smokers. Participants experienced significant technology problems during the study. Some participants received duplicate text messages at least once during the trial; others failed to receive some program messages. Neither receiving duplicate messages (χ(2)(1)=0.12, P=.73), or missing 5 or more program messages (χ(2)(1)=0.75, P=.39) negatively affected quitting rates.ConclusionsAlthough the study was not powered to detect statistically significant differences, as the primary aim was to provide estimates of effect size that could be used to better inform a power analysis for a larger trial, findings provide optimism that SMS Turkey may be able to affect quitting rates in environments with high smoking prevalence, such as Ankara, Turkey. The SMS Turkey software program did not work as well as it did 2 years previous. The system will need to be updated to maintain software compatibility with ongoing technology evolution.Trial registrationClinicaltrials.gov NCT00912795 http://clinicaltrials.gov/ct2/show/NCT00912795 (Archived by WebCite® at http://www.webcitation.org/6Ch1cIA8l).

Project description:BackgroundMobile health strategies for smoking cessation (eg, SMS text messaging-based interventions) have been shown to be effective in helping smokers quit. However, further research is needed to better understand user experiences with these platforms.ObjectiveThis qualitative study aims to explore the experiences of real-world users of a publicly available smoking cessation program (SmokefreeTXT).MethodsSemistructured phone interviews were conducted with 36 SmokefreeTXT users between March and July 2014. Of these 36 participants, 50% (18/36) of participants completed the SmokefreeTXT program (ie, did not opt out of the program before the 6- to 8-week completion period), and 50% (18/36) did not complete the program (ie, requested to opt out of the program before the completion period). Interview questions focused on smoking behaviors, quitting history, opinions on the program's content and structure, answering assessment questions, using keywords, reasons for opting out, and perceived usefulness of the program for quitting smoking. A thematic content analysis was conducted, with a focus on themes to increase program engagement and optimization.ResultsThe findings highlighted features of the program that participants found beneficial, as well as some elements that showed opportunities for improvement to boost program retention and successful cessation. Specifically, most participants found the SmokefreeTXT program to be convenient and supportive of cessation; however, some found the messages to be repetitive and reported a desire for more flexibility based on their readiness to quit and cessation progress. We also found that program completion did not necessarily indicate successful smoking cessation and that program opt out, which might be interpreted as a less positive outcome, may occur because of successful cessation. Finally, several participants reported using SmokefreeTXT together with other evidence-based cessation methods or non-evidence-based strategies.ConclusionsQualitative interviews with real-world SmokefreeTXT users showed high program acceptability, engagement with program features, and perceived utility for smoking cessation. Our findings directly informed several program updates, such as adding an adaptive quit date feature and offering supplemental information on live support services for users who prefer human interaction during the cessation process. The study has implications for other digital tobacco cessation interventions and highlights important topics that warrant future research, such as the relationship between program engagement (eg, opt out and retention) and successful cessation.

Project description:INTRODUCTION:Mobile health interventions have the potential to promote risk factor management and lifestyle modification, and are a particularly attractive approach for scaling across healthcare systems with limited resources. We are conducting two randomised trials to evaluate the efficacy of text message-based health messages in improving secondary coronary heart disease (CHD) prevention among patients with or without diabetes. METHODS AND ANALYSIS:The Cardiovascular Health And Text Messaging (CHAT) Study and the CHAT-Diabetes Mellitus (CHAT-DM) Study are multicentre, single-blind, randomised controlled trials of text messaging versus standard treatment with 6 months of follow-up conducted in 37 hospitals throughout 17 provinces in China. The intervention group receives six text messages per week which target blood pressure control, medication adherence, physical activity, smoking cessation (when appropriate), glucose monitoring and lifestyle recommendations including diet (in CHAT-DM). The text messages were developed based on behavioural change techniques, using models such as the information-motivation-behavioural skills model, goal setting and provision of social support. A total sample size of 800 patients would be adequate for CHAT Study and sample size of 500 patients would be adequate for the CHAT-DM Study. In CHAT, the primary outcome is the change in systolic blood pressure (SBP) at 6 months. Secondary outcomes include a change in proportion of patients achieving a SBP <140 mm Hg, low-density lipoprotein cholesterol (LDL-C), physical activity, medication adherence, body mass index (BMI) and smoking cessation. In CHAT-DM, the primary outcome is the change in glycaemic haemoglobin (HbA1C) at 6 months. Secondary outcomes include a change in the proportion of patients achieving HbA1C<7%, fasting blood glucose, SBP, LDL-C, BMI, physical activity and medication adherence. ETHICS AND DISSEMINATION:The central ethics committee at the China National Center for Cardiovascular Disease and the Yale University Institutional Review Board approved the CHAT and CHAT-DM studies. Results will be disseminated via usual scientific forums including peer-reviewed publications. TRIAL REGISTRATION NUMBER:CHAT (NCT02888769) and CHAT-DM (NCT02883842); Pre-results.

Project description:BackgroundMobile health tools such as text messaging programs can support smoking cessation. However, high rates of disengagement from these tools decrease their effectiveness.ObjectiveThe purpose of this study was to identify user characteristics associated with retention in an adult text messaging smoking cessation intervention.MethodsAdults initiating a quit attempt using the publicly available program SmokefreeTXT between March 6 and June 21, 2016 (n=6215), were included. Data were collected to assess nicotine dependence, frequency of being around other smokers, time of the day for cigarette cravings, extrinsic and intrinsic motivation to quit smoking, confidence in quitting, and long-term intention to be smoke free. Multivariable survival analysis modeling for time to opt out was conducted to identify characteristics associated with opting out over the course of the intervention, adjusting for age, sex, and smoking frequency, reset of the quit date by the user, and the number of days enrolled before initiating the quit attempt. Among those who opted out, multivariable multinomial logistic regression analysis was used to identify predictors of opting out early (within 3 days and between 4 and 7 days into the quit attempt) compared to opting out late (more than 7 days into the quit attempt), adjusting for the same confounders.ResultsSurvival analyses indicated that younger age, female sex, higher levels of nicotine dependence, lower intention to be smoke free, and enrolling in SmokefreeTXT ≤1 week before initiating the quit attempt were associated with an increased risk of opting out. For example, users who smoked within 5 minutes of waking up were 1.17 times more likely to opt out than those who smoked more than 5 minutes after waking up (95% CI 1.01-1.35). Among users who opted out from SmokefreeTXT, logistic regression modeling indicated that compared to users who were never or rarely around other smokers, those who were sometimes around other smokers had 1.96 times more likely to opt out within the first 3 days of the quit attempt (95% CI 1.18-3.25). In addition, compared to users with high levels of long-term quit intention, users with lower levels of intention had 1.80 times the odds of opting out between 4 and 7 days into the quit attempt (95% CI 1.02-3.18). Users who reset their quit date after initiating a quit attempt were less likely to opt out at either time point compared with those who did not reset their quit date.ConclusionsSeveral user characteristics are associated with retention in an adult text messaging smoking cessation program. These results provide guidance on potential characteristics that should be addressed in future text messaging smoking cessation programs. Providing additional support to users with these characteristics may increase retention in text messaging programs and ultimately lead to smoking cessation.

Dataset Information

Use of a Machine Learning Program to Correctly Triage Incoming Text Messaging Replies From a Cardiovascular Text-Based Secondary Prevention Program: Feasibility Study.

Publications

Use of a Machine Learning Program to Correctly Triage Incoming Text Messaging Replies From a Cardiovascular Text-Based Secondary Prevention Program: Feasibility Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets