Dataset Information

Detecting premature departure in online text-based counseling using logic-based pattern matching.

ABSTRACT:

Background

More so than face-to-face counseling, users of online text-based services might drop out from a session before establishing a clear closure or expressing the intention to leave. Such premature departure may be indicative of heightened risk or dissatisfaction with the service or counselor. However, there is no systematic way to identify this understudied phenomenon.

Purpose

This study has two objectives. First, we developed a set of rules and used logic-based pattern matching techniques to systematically identify premature departures in an online text-based counseling service. Second, we validated the importance of premature departure by examining its association with user satisfaction. We hypothesized that the users who rated the session as less helpful were more likely to have departed prematurely.

Method

We developed and tested a classification model using a sample of 575 human-annotated sessions from an online text-based counseling platform. We used 80% of the dataset to train and develop the model and 20% of the dataset to evaluate the model performance. We further applied the model to the full dataset (34,821 sessions). We compared user satisfaction between premature departure and completed sessions based on data from a post-session survey.

Results

The resulting model achieved 97% and 92% F1 score in detecting premature departure cases in the training and test sets, respectively, suggesting it is highly consistent with the judgment of human coders. When applied to the full dataset, the model classified 15,150 (43.5%) sessions as premature departure and the remaining 19,671 (56.5%) as completed sessions. Completed cases (15.2%) were more likely to fill the post-chat survey than premature departure cases (4.0%). Premature departure was significantly associated with lower perceived helpfulness and effectiveness in distress reduction.

Conclusions

The model is the first that systematically and accurately identifies premature departure in online text-based counseling. It can be readily modified and transferred to other contexts for the purpose of risk mitigation and service evaluation and improvement.

SUBMITTER: Xu Y

PROVIDER: S-EPMC8632835 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:Young people aged 10-24 years are at the highest risk for mental health problems and are the least likely to seek professional treatment. Owing to this population's high consumption of internet content, electronic mental (e-mental) health services have increased globally, with an aim to address barriers to treatment. Many of these services use text-based online counseling (TBOC), which shows promising results in supporting young people but also greater variance in outcomes compared with adult comparators. OBJECTIVE:This pilot study qualitatively explored the characteristics of users aged 15-25 years accessing TBOC services, their motivations for access, and their perceptions about factors believed to influence the effectiveness of these modalities. METHODS:E-surveys were administered naturalistically to 100 young service users aged 15-25 years who accessed webchat and email counseling services via an Australian e-mental health service. Thematic analysis of qualitative themes and quantitative descriptive and proportional data presented in electronic surveys were examined across the areas of user characteristics, motivations for selecting TBOC modalities, and their perceptions of TBOC effectiveness. RESULTS:Participants were predominately female high school students of Caucasian or European descent from middle socioeconomic status, living with their parents in major cities. Four domains and various themes and subthemes were related to participants' reasons for accessing TBOC and perceptions of its effectiveness: user characteristics (ie, physical and mental health syndrome and perceived social difficulties), selection factors (ie, safety, avoidance motivation, accessibility, and expectation), factors perceived to increase effectiveness (ie, general therapeutic benefits, positive modality and service factors, and persisting with counseling to increase benefit), and factors perceived to decrease effectiveness (ie, negative modality and service factors, and persisting with counseling despite benefit). CONCLUSIONS:Participants were motivated to use TBOC to increase their sense of safety in response to negative perceptions of their social skills and the response of the online counsellor to their presenting problem. By using TBOC services, they also sought to improve their access to mental health services that better met their expectations. Factors that increased effectiveness of TBOC were the counsellor's interpersonal skills, use of text-based communication, and persisting with beneficial counseling sessions. Factors that reduced TBOC effectiveness were poor timeliness in response to service requests, experiencing no change in their presenting problem, not knowing what postcounseling action to take, and persisting with ineffective counseling sessions.

Project description:BACKGROUND:Population-based studies show that the risk of mental ill health is highest among young people aged 10 to 24 years, who are also the least likely to seek professional treatment because of a number of barriers. Electronic mental (e-mental) health services have been advocated as a method for decreasing these barriers for young people, among which text-based online counseling (TBOC) is a primary intervention used at many youth-oriented services. Although TBOC has shown promising results, its outcome variance is greater in comparison with other electronic interventions and adult user groups. OBJECTIVE:This pilot study aimed to explore and confirm e-mental health professional's perspectives about various domains and themes related to young service users' (YSUs) motivations for accessing TBOC services and factors related to higher and lower effectiveness on these modalities. METHODS:Participants were 9 e-mental health professionals who were interviewed individually and in focus groups using a semistructured interview. Thematic analysis of qualitative themes from interview transcripts was examined across the areas of YSU motivations for access and factors that increase and decrease TBOC effectiveness. RESULTS:A total of 4 domains and various subthemes were confirmed and identified to be related to YSUs' characteristics, motivations for accessing TBOC, and moderators of service effectiveness: user characteristics (ie, prior negative help-seeking experience, mental health syndrome, limited social support, and perceived social difficulties), selection factors (ie, safety, avoidance motivation, accessibility, and expectation), and factors perceived to increase effectiveness (ie, general therapeutic benefits, positive service-modality factors, and persisting with counseling despite substantial benefit) and decrease effectiveness (ie, negative service-modality factors). CONCLUSIONS:Participants perceived YSUs to have polarized expectations of TBOC effectiveness and be motivated by service accessibility and safety, in response to several help-seeking concerns. Factors increasing TBOC effectiveness were using text-based communication, the online counselor's interpersonal skills and use of self-management and crisis-support strategies, and working with less complex presenting problems or facilitating access to more intensive support. Factors decreasing TBOC effectiveness were working with more complex problems owing to challenges with assessment, the slow pace of text communication, lack of nonverbal conversational cues, and environmental and connectivity issues. Other factors were using ineffective techniques (eg, poor goal setting, focusing, and postcounseling direction) that produced only short-term outcomes, poor timeliness in responding to service requests, rupture in rapport from managing service boundaries, and low YSU readiness and motivation.