Dataset Information

Accuracy of a Chatbot (Ada) in the Diagnosis of Mental Disorders: Comparative Case Study With Lay and Expert Users.

ABSTRACT:

Background

Health apps for the screening and diagnosis of mental disorders have emerged in recent years on various levels (eg, patients, practitioners, and public health system). However, the diagnostic quality of these apps has not been (sufficiently) tested so far.

Objective

The objective of this pilot study was to investigate the diagnostic quality of a health app for a broad spectrum of mental disorders and its dependency on expert knowledge.

Methods

Two psychotherapists, two psychology students, and two laypersons each read 20 case vignettes with a broad spectrum of mental disorders. They used a health app (Ada-Your Health Guide) to get a diagnosis by entering the symptoms. Interrater reliabilities were computed between the diagnoses of the case vignettes and the results of the app for each user group.

Results

Overall, there was a moderate diagnostic agreement (kappa=0.64) between the results of the app and the case vignettes for mental disorders in adulthood and a low diagnostic agreement (kappa=0.40) for mental disorders in childhood and adolescence. When psychotherapists applied the app, there was a good diagnostic agreement (kappa=0.78) regarding mental disorders in adulthood. The diagnostic agreement was moderate (kappa=0.55/0.60) for students and laypersons. For mental disorders in childhood and adolescence, a moderate diagnostic quality was found when psychotherapists (kappa=0.53) and students (kappa=0.41) used the app, whereas the quality was low for laypersons (kappa=0.29). On average, the app required 34 questions to be answered and 7 min to complete.

Conclusions

The health app investigated here can represent an efficient diagnostic screening or help function for mental disorders in adulthood and has the potential to support especially diagnosticians in their work in various ways. The results of this pilot study provide a first indication that the diagnostic accuracy is user dependent and improvements in the app are needed especially for mental disorders in childhood and adolescence.

SUBMITTER: Jungmann SM

PROVIDER: S-EPMC6914276 | biostudies-literature | 2019 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Accuracy of a Chatbot (Ada) in the Diagnosis of Mental Disorders: Comparative Case Study With Lay and Expert Users.

Jungmann Stefanie Maria SM Klan Timo T Kuhn Sebastian S Jungmann Florian F

JMIR formative research 20191029 4

<h4>Background</h4>Health apps for the screening and diagnosis of mental disorders have emerged in recent years on various levels (eg, patients, practitioners, and public health system). However, the diagnostic quality of these apps has not been (sufficiently) tested so far.<h4>Objective</h4>The objective of this pilot study was to investigate the diagnostic quality of a health app for a broad spectrum of mental disorders and its dependency on expert knowledge.<h4>Methods</h4>Two psychotherapist ...[more]

PMID: 31663858

Similar Datasets

Project description:ImportanceNatural language processing tools, such as ChatGPT (generative pretrained transformer, hereafter referred to as chatbot), have the potential to radically enhance the accessibility of medical information for health professionals and patients. Assessing the safety and efficacy of these tools in answering physician-generated questions is critical to determining their suitability in clinical settings, facilitating complex decision-making, and optimizing health care efficiency.ObjectiveTo assess the accuracy and comprehensiveness of chatbot-generated responses to physician-developed medical queries, highlighting the reliability and limitations of artificial intelligence-generated medical information.Design, setting, and participantsThirty-three physicians across 17 specialties generated 284 medical questions that they subjectively classified as easy, medium, or hard with either binary (yes or no) or descriptive answers. The physicians then graded the chatbot-generated answers to these questions for accuracy (6-point Likert scale with 1 being completely incorrect and 6 being completely correct) and completeness (3-point Likert scale, with 1 being incomplete and 3 being complete plus additional context). Scores were summarized with descriptive statistics and compared using the Mann-Whitney U test or the Kruskal-Wallis test. The study (including data analysis) was conducted from January to May 2023.Main outcomes and measuresAccuracy, completeness, and consistency over time and between 2 different versions (GPT-3.5 and GPT-4) of chatbot-generated medical responses.ResultsAcross all questions (n = 284) generated by 33 physicians (31 faculty members and 2 recent graduates from residency or fellowship programs) across 17 specialties, the median accuracy score was 5.5 (IQR, 4.0-6.0) (between almost completely and complete correct) with a mean (SD) score of 4.8 (1.6) (between mostly and almost completely correct). The median completeness score was 3.0 (IQR, 2.0-3.0) (complete and comprehensive) with a mean (SD) score of 2.5 (0.7). For questions rated easy, medium, and hard, the median accuracy scores were 6.0 (IQR, 5.0-6.0), 5.5 (IQR, 5.0-6.0), and 5.0 (IQR, 4.0-6.0), respectively (mean [SD] scores were 5.0 [1.5], 4.7 [1.7], and 4.6 [1.6], respectively; P = .05). Accuracy scores for binary and descriptive questions were similar (median score, 6.0 [IQR, 4.0-6.0] vs 5.0 [IQR, 3.4-6.0]; mean [SD] score, 4.9 [1.6] vs 4.7 [1.6]; P = .07). Of 36 questions with scores of 1.0 to 2.0, 34 were requeried or regraded 8 to 17 days later with substantial improvement (median score 2.0 [IQR, 1.0-3.0] vs 4.0 [IQR, 2.0-5.3]; P < .01). A subset of questions, regardless of initial scores (version 3.5), were regenerated and rescored using version 4 with improvement (mean accuracy [SD] score, 5.2 [1.5] vs 5.7 [0.8]; median score, 6.0 [IQR, 5.0-6.0] for original and 6.0 [IQR, 6.0-6.0] for rescored; P = .002).Conclusions and relevanceIn this cross-sectional study, chatbot generated largely accurate information to diverse medical queries as judged by academic physician specialists with improvement over time, although it had important limitations. Further research and model development are needed to correct inaccuracies and for validation.

Project description:BackgroundConversational user interfaces, or chatbots, are becoming more popular in the realm of digital health and well-being. While many studies focus on measuring the cause or effect of a digital intervention on people's health and well-being (outcomes), there is a need to understand how users really engage and use a digital intervention in the real world.ObjectiveIn this study, we examine the user logs of a mental well-being chatbot called ChatPal, which is based on the concept of positive psychology. The aim of this research is to analyze the log data from the chatbot to provide insight into usage patterns, the different types of users using clustering, and associations between the usage of the app's features.MethodsLog data from ChatPal was analyzed to explore usage. A number of user characteristics including user tenure, unique days, mood logs recorded, conversations accessed, and total number of interactions were used with k-means clustering to identify user archetypes. Association rule mining was used to explore links between conversations.ResultsChatPal log data revealed 579 individuals older than 18 years used the app with most users being female (n=387, 67%). User interactions peaked around breakfast, lunchtime, and early evening. Clustering revealed 3 groups including "abandoning users" (n=473), "sporadic users" (n=93), and "frequent transient users" (n=13). Each cluster had distinct usage characteristics, and the features were significantly different (P<.001) across each group. While all conversations within the chatbot were accessed at least once by users, the "treat yourself like a friend" conversation was the most popular, which was accessed by 29% (n=168) of users. However, only 11.7% (n=68) of users repeated this exercise more than once. Analysis of transitions between conversations revealed strong links between "treat yourself like a friend," "soothing touch," and "thoughts diary" among others. Association rule mining confirmed these 3 conversations as having the strongest linkages and suggested other associations between the co-use of chatbot features.ConclusionsThis study has provided insight into the types of people using the ChatPal chatbot, patterns of use, and associations between the usage of the app's features, which can be used to further develop the app by considering the features most accessed by users.

Dataset Information

Accuracy of a Chatbot (Ada) in the Diagnosis of Mental Disorders: Comparative Case Study With Lay and Expert Users.

Background

Objective

Methods

Results

Conclusions

Publications

Accuracy of a Chatbot (Ada) in the Diagnosis of Mental Disorders: Comparative Case Study With Lay and Expert Users.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets