Dataset Information

Evaluation of Four Artificial Intelligence-Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study.

ABSTRACT:

Background

Consumer-oriented mobile self-diagnosis apps have been developed using undisclosed algorithms, presumably based on machine learning and other artificial intelligence (AI) technologies. The US Food and Drug Administration now discerns apps with learning AI algorithms from those with stable ones and treats the former as medical devices. To the author's knowledge, no self-diagnosis app testing has been performed in the field of ophthalmology so far.

Objective

The objective of this study was to test apps that were previously mentioned in the scientific literature on a set of diagnoses in a deliberate time interval, comparing the results and looking for differences that hint at "nonlocked" learning algorithms.

Methods

Four apps from the literature were chosen (Ada, Babylon, Buoy, and Your.MD). A set of three ophthalmology diagnoses (glaucoma, retinal tear, dry eye syndrome) representing three levels of urgency was used to simultaneously test the apps' diagnostic efficiency and treatment recommendations in this specialty. Two years was the chosen time interval between the tests (2018 and 2020). Scores were awarded by one evaluating physician using a defined scheme.

Results

Two apps (Ada and Your.MD) received significantly higher scores than the other two. All apps either worsened in their results between 2018 and 2020 or remained unchanged at a low level. The variation in the results over time indicates "nonlocked" learning algorithms using AI technologies. None of the apps provided correct diagnoses and treatment recommendations for all three diagnoses in 2020. Two apps (Babylon and Your.MD) asked significantly fewer questions than the other two (P<.001).

Conclusions

"Nonlocked" algorithms are used by self-diagnosis apps. The diagnostic efficiency of the tested apps seems to worsen over time, with some apps being more capable than others. Systematic studies on a wider scale are necessary for health care providers and patients to correctly assess the safety and efficacy of such apps and for correct classification by health care regulating authorities.

SUBMITTER: Cirkovic A

PROVIDER: S-EPMC7748958 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundSymptom checkers are digital tools assisting laypersons in self-assessing the urgency and potential causes of their medical complaints. They are widely used but face concerns from both patients and health care professionals, especially regarding their accuracy. A 2015 landmark study substantiated these concerns using case vignettes to demonstrate that symptom checkers commonly err in their triage assessment.ObjectiveThis study aims to revisit the landmark index study to investigate whether and how symptom checkers' capabilities have evolved since 2015 and how they currently compare with laypersons' stand-alone triage appraisal.MethodsIn early 2020, we searched for smartphone and web-based applications providing triage advice. We evaluated these apps on the same 45 case vignettes as the index study. Using descriptive statistics, we compared our findings with those of the index study and with publicly available data on laypersons' triage capability.ResultsWe retrieved 22 symptom checkers providing triage advice. The median triage accuracy in 2020 (55.8%, IQR 15.1%) was close to that in 2015 (59.1%, IQR 15.5%). The apps in 2020 were less risk averse (odds 1.11:1, the ratio of overtriage errors to undertriage errors) than those in 2015 (odds 2.82:1), missing >40% of emergencies. Few apps outperformed laypersons in either deciding whether emergency care was required or whether self-care was sufficient. No apps outperformed the laypersons on both decisions.ConclusionsTriage performance of symptom checkers has, on average, not improved over the course of 5 years. It decreased in 2 use cases (advice on when emergency care is required and when no health care is needed for the moment). However, triage capability varies widely within the sample of symptom checkers. Whether it is beneficial to seek advice from symptom checkers depends on the app chosen and on the specific question to be answered. Future research should develop resources (eg, case vignette repositories) to audit the capabilities of symptom checkers continuously and independently and provide guidance on when and to whom they should be recommended.

Project description:Background: Emotional intelligence (EI) is thought to play a significant role in professional and academic success. EI is important for medical personnel to cope with highly stressful circumstances during clinical and academic settings. The present prospective follow-up study intends to evaluate the changes in EI and their correlates among medical undergraduates over a five-year period.Methods: Data were collected in 2015 and 2020 at the Faculty of Medicine, University of Colombo, Sri Lanka. EI was assessed using the validated 33-item self-assessment tool, Schutte Self-Report Emotional Intelligence Test (SSEIT). In addition, socio-demographic details, students' involvement in extracurricular-activities during undergraduate life, students' satisfaction regarding the choice of studying medicine and plans to do postgraduate studies were also evaluated. A multiple-regression analysis was conducted among all students using percentage change in EI score as the continuous dependent variable, together with other independent variables (plan to do postgraduate studies, satisfaction in choice of medicine and extracurricular-activities).Results: Sample size was 170 (response rates-96.6%), with 41.2% males (n?=?70). Mean EI scores at baseline among all students was 122.7?±?11.6, and it had significantly increased at follow-up to 128.9?±?11.2 (p?<? 0.001). This significant increase was independently observed in both males (122.1?±?12.2 vs. 130.0?±?12.4, p?<? 0.001) and females (123.1?±?11.1 vs. 128.2?±?10.3, p?=?0.001). During follow-up, an increase in EI score was observed in students of all religions and ethnicities. Mean EI score also increased in all categories of monthly income, irrespective of the employment status or attainment of higher education of either parent. An increase in mean EI score during follow-up was observed in students irrespective of their engagement in or number of extracurricular-activities, they were involved. In the multiple regression analysis, being satisfied regarding their choice of the medical undergraduate programme (OR:11.75, p?=?0.001) was the only significant factor associated with the percentage change in EI score.Conclusion: EI in this group significantly improved over 5-years of follow-up and was independent of gender, religion, ethnicity, socio-economic parameters and academic performance. Satisfaction in the chosen field was a significant predictor of the overall change in EI. Future studies are needed to identify and measure factors responsible for improvement in EI among medical undergraduates.

Project description:ObjectiveT cell immunity plays an important role in anti-tumor effects and immunosuppression often leads to the development and relapse of cancer. This study aimed to investigate the effect of T cell numbers on the long-term prognosis of patients with hepatocellular carcinoma (HCC) and construct an artificial neural network (ANN) model to evaluate its prognostic value.MethodsWe enrolled 3,427 patients with HCC at Beijing Ditan Hospital, Capital Medical University, and randomly divided them into two groups of 1,861 and 809 patients as the training and validation sets, respectively. Cox regression analysis was used to screen for independent risk factors of survival in patients with HCC. These factors were used to build an ANN model using Python. Concordance index, calibration curve, and decision curve analysis were used to evaluate the model performance.ResultsThe 1-year, 3-year, 5-year, and 10-year cumulative overall survival (OS) rates were 66.9%, 45.7%, 34.9%, and 22.6%, respectively. Cox multivariate regression analysis showed that age, white blood cell count, creatinine, total bilirubin, γ-GGT, LDH, tumor size ≥ 5 cm, tumor number ≥ 2, portal vein tumor thrombus, and AFP ≥ 400 ng/ml were independent risk factors for long-term survival in HCC. Antiviral therapy, albumin, T cell, and CD8 T cell counts were independent protective factors. An ANN model was developed for long-term survival. The areas under the receiver operating characteristic (ROC) curve of 1-year, 3-year, and 5-year OS rates by ANNs were 0.838, 0.833, and 0.843, respectively, which were higher than those of the Barcelona Clinic Liver Cancer (BCLC), tumor node metastasis (TNM), Okuda, Chinese University Prognostic Index (CUPI), Cancer of the Liver Italian Program (CLIP), Japan Integrated Staging (JIS), and albumin-bilirubin (ALBI) models (P < 0.0001). According to the ANN model scores, all patients were divided into high-, middle-, and low-risk groups. Compared with low-risk patients, the hazard ratios of 5-year OS of the high-risk group were 8.11 (95% CI: 7.0-9.4) and 6.13 (95% CI: 4.28-8.79) (P<0.0001) in the training and validation sets, respectively.ConclusionHigh levels of circulating T cells and CD8 + T cells in peripheral blood may benefit the long-term survival of patients with HCC. The ANN model has a good individual prediction performance, which can be used to assess the prognosis of HCC and lay the foundation for the implementation of precision treatment in the future.