Accuracy of a Popular Online Symptom Checker for Ophthalmic Diagnoses.
Ontology highlight
ABSTRACT: Importance
Because more patients are presenting with self-guided research of symptoms, it is important to assess the capabilities and limitations of these available health information tools.Objective
To determine the accuracy of the most popular online symptom checker for ophthalmic diagnoses.Design, setting, and participants
In a cross-sectional study, 42 validated clinical vignettes of ophthalmic symptoms were generated and distilled to their core presenting symptoms. Cases were entered into WebMD symptom checker by both medically trained and nonmedically trained personnel blinded to the diagnosis. Output from the symptom checker, including the number of symptoms, ranking and list of diagnoses, and triage urgency were recorded. The study was conducted on October 13, 2017. Analysis was performed between October 15, 2017, and April 30, 2018.Main outcomes and measures
Accuracy of the top 3 diagnoses generated by the online symptom checker.Results
The mean (SD) number of symptoms entered was 3.6 (1.6) (range, 1-8). The median (SD) number of diagnoses generated by the symptom checker was 26.8 (21.8) (range, 1-99). The primary diagnosis by the symptom checker was correct in 11 of 42 (26%; 95% CI, 12%-40%) cases. The correct diagnosis was included in the online symptom checker's top 3 diagnoses in 16 of 42 (38%; 95% CI, 25%-56%) cases. The correct diagnosis was not included in the symptom checker's list in 18 of 42 (43%; 95% CI, 32%-63%) cases. Triage urgency based on the top diagnosis was appropriate in 7 of 18 (39%; 95% CI, 14%-64%) emergent cases and 21 of 24 (88%; 95% CI, 73%-100%) nonemergent cases. Interuser variability for the correct diagnosis being in the top 3 listed was at least moderate (Cohen κ = 0.74; 95% CI, 0.54-0.95).Conclusions and relevance
The most popular online symptom checker may arrive at the correct clinical diagnosis for ophthalmic conditions, but a substantial proportion of diagnoses may not be captured. These findings suggest that further research to reflect the real-life application of internet diagnostic resources is required.
SUBMITTER: Shen C
PROVIDER: S-EPMC6567837 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Digital and online symptom checkers are an increasingly adopted class of health technologies that enable patients to input their symptoms and biodata to produce a set of likely diagnoses and associated triage advice. However, concerns regarding the accuracy and safety of these symptom checkers have been raised. This systematic review evaluates the accuracy of symptom checkers in providing diagnoses and appropriate triage advice. MEDLINE and Web of Science were searched for studies that used either real or simulated patients to evaluate online or digital symptom checkers. The primary outcomes were the diagnostic and triage accuracy of the symptom checkers. The QUADAS-2 tool was used to assess study quality. Of the 177 studies retrieved, 10 studies met the inclusion criteria. Researchers evaluated the accuracy of symptom checkers using a variety of medical conditions, including ophthalmological conditions, inflammatory arthritides and HIV. A total of 50% of the studies recruited real patients, while the remainder used simulated cases. The diagnostic accuracy of the primary diagnosis was low across included studies (range: 19-37.9%) and varied between individual symptom checkers, despite consistent symptom data input. Triage accuracy (range: 48.8-90.1%) was typically higher than diagnostic accuracy. Overall, the diagnostic and triage accuracy of symptom checkers are variable and of low accuracy. Given the increasing push towards adopting this class of technologies across numerous health systems, this study demonstrates that reliance upon symptom checkers could pose significant patient safety hazards. Large-scale primary studies, based upon real-world data, are warranted to demonstrate the adequate performance of these technologies in a manner that is non-inferior to current best practices. Moreover, an urgent assessment of how these systems are regulated and implemented is required.
| S-EPMC9385087 | biostudies-literature
Project description:BackgroundSymptom checkers are digital tools assisting laypersons in self-assessing the urgency and potential causes of their medical complaints. They are widely used but face concerns from both patients and health care professionals, especially regarding their accuracy. A 2015 landmark study substantiated these concerns using case vignettes to demonstrate that symptom checkers commonly err in their triage assessment.ObjectiveThis study aims to revisit the landmark index study to investigate whether and how symptom checkers' capabilities have evolved since 2015 and how they currently compare with laypersons' stand-alone triage appraisal.MethodsIn early 2020, we searched for smartphone and web-based applications providing triage advice. We evaluated these apps on the same 45 case vignettes as the index study. Using descriptive statistics, we compared our findings with those of the index study and with publicly available data on laypersons' triage capability.ResultsWe retrieved 22 symptom checkers providing triage advice. The median triage accuracy in 2020 (55.8%, IQR 15.1%) was close to that in 2015 (59.1%, IQR 15.5%). The apps in 2020 were less risk averse (odds 1.11:1, the ratio of overtriage errors to undertriage errors) than those in 2015 (odds 2.82:1), missing >40% of emergencies. Few apps outperformed laypersons in either deciding whether emergency care was required or whether self-care was sufficient. No apps outperformed the laypersons on both decisions.ConclusionsTriage performance of symptom checkers has, on average, not improved over the course of 5 years. It decreased in 2 use cases (advice on when emergency care is required and when no health care is needed for the moment). However, triage capability varies widely within the sample of symptom checkers. Whether it is beneficial to seek advice from symptom checkers depends on the app chosen and on the specific question to be answered. Future research should develop resources (eg, case vignette repositories) to audit the capabilities of symptom checkers continuously and independently and provide guidance on when and to whom they should be recommended.
| S-EPMC9131144 | biostudies-literature
Project description:BackgroundOnline symptom checkers are digital health solutions that provide a differential diagnosis based on a user's symptoms. During the coronavirus disease 2019 (COVID-19) pandemic, symptom checkers have become increasingly important due to physical distance constraints and reduced access to in-person medical consultations. Furthermore, various symptom checkers specialised in the assessment of COVID-19 infection have been produced.ObjectivesAssess the correlation between COVID-19 risk assessments from an online symptom checker and current trends in COVID-19 infections. Analyse whether those correlations are reflective of various country-wise quality of life measures. Lastly, determine whether the trends found in symptom checker assessments predict or lag relative to those of the COVID-19 infections.Materials and methodsIn this study, we compile the outcomes of COVID-19 risk assessments provided by the symptom checker Symptoma (www.symptoma.com) in 18 countries with suitably large user bases. We analyse this dataset's spatial and temporal features compared to the number of newly confirmed COVID-19 cases published by the respective countries.ResultsWe find an average correlation of 0.342 between the number of Symptoma users assessed to have a high risk of a COVID-19 infection and the official COVID-19 infection numbers. Further, we show a significant relationship between that correlation and the self-reported health of a country. Lastly, we find that the symptom checker is, on average, ahead (median +3 days) of the official infection numbers for most countries.ConclusionWe show that online symptom checkers can capture the national-level trends in coronavirus infections. As such, they provide a valuable and unique information source in policymaking against pandemics, unrestricted by conventional resources.
| S-EPMC9917242 | biostudies-literature
Project description:BackgroundSymptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions.ObjectiveThe aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED).MethodsWe recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker's diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model.ResultsA total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although "satisfaction" and "enjoyment" were rated low.ConclusionsThis study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.
| S-EPMC9531004 | biostudies-literature
Project description:In light of growing interest in using emerging large language models (LLMs) for self-diagnosis, we systematically assessed the performance of ChatGPT-3.5, ChatGPT-4.0, and Google Bard in delivering proficient responses to 37 common inquiries regarding ocular symptoms. Responses were masked, randomly shuffled, and then graded by three consultant-level ophthalmologists for accuracy (poor, borderline, good) and comprehensiveness. Additionally, we evaluated the self-awareness capabilities (ability to self-check and self-correct) of the LLM-Chatbots. 89.2% of ChatGPT-4.0 responses were 'good'-rated, outperforming ChatGPT-3.5 (59.5%) and Google Bard (40.5%) significantly (all p < 0.001). All three LLM-Chatbots showed optimal mean comprehensiveness scores as well (ranging from 4.6 to 4.7 out of 5). However, they exhibited subpar to moderate self-awareness capabilities. Our study underscores the potential of ChatGPT-4.0 in delivering accurate and comprehensive responses to ocular symptom inquiries. Future rigorous validation of their performance is crucial to ensure their reliability and appropriateness for actual clinical use.
| S-EPMC10616302 | biostudies-literature
Project description:Patient online health searching is now commonplace, however, the accuracy of patient generated differentials for new symptoms and potential for patient anxiety are concerns. We aimed primarily to determine the accuracy of patient generated differentials for new symptoms with and without online searching, and secondarily, to evaluate the impact of searching on anxiety levels. In the waiting room prior to seeing a clinician, 300 patients with new symptoms were randomly assigned 1:1:1 to Google searching with health related features including a symptom search tool vs Google searching with health related features disabled vs no searching. Participants were 18 years or older and presenting to the emergency department of an urban academic medical center with new low-acuity symptoms that were not due to exacerbation of a chronic condition. Search groups received access on a tablet/smartphone to Google searching with or without health related features. Both search groups could access any websites; health related features led the patient to common diagnoses and physician-validated information. The primary outcome was accuracy of the patient generated differential assessed by matching at least two of the top three diagnoses on the clinician's differential. A secondary outcome was anxiety by a visual analogue scale. Patients were a median of 33.1 (IQI 26.2-45.9) years old, 60% women, 63% black, 82% had a high school education or less, and 45.7% reported having performed an online search prior to presentation. Search group patients spent a median of 3.82 (2.53-5.72) minutes searching online. Similar proportions of patients in each group matched at least two of three clinician diagnoses: 27.0% and 28.3% for Google searching with and without health related features vs 23.8% in the no search group. Patients in the search groups had a similar odds of matching ≥2/3 diagnoses as the no search group [OR (95% CI): 1.23 (0.70-2.13), p = 0.47]. Anxiety was unchanged with online searching. In conclusion, brief online searching in the waiting room did not improve accuracy of patient generated differential diagnoses for new symptoms. The absence of an increase in patient anxiety provides reassurance for subsequent work to refine and investigate online symptom search tools.
| S-EPMC6848180 | biostudies-literature
Project description:ObjectivesThe aims of our study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems.SettingPublicly available symptom checkers for patient use.ParticipantsPublicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes were developed and systematically run through each system by a non-clinical researcher.Primary and secondary outcome measuresSystem accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette.ResultsTwelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). Safety of disposition advice decreased with condition urgency (being 71.8% for emergency cases vs 87.3% for non-urgent cases). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems.ConclusionsThere is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.
| S-EPMC8282353 | biostudies-literature
Project description:BackgroundThe diagnosis of inflammatory rheumatic diseases (IRDs) is often delayed due to unspecific symptoms and a shortage of rheumatologists. Digital diagnostic decision support systems (DDSSs) have the potential to expedite diagnosis and help patients navigate the health care system more efficiently.ObjectiveThe aim of this study was to assess the diagnostic accuracy of a mobile artificial intelligence (AI)-based symptom checker (Ada) and a web-based self-referral tool (Rheport) regarding IRDs.MethodsA prospective, multicenter, open-label, crossover randomized controlled trial was conducted with patients newly presenting to 3 rheumatology centers. Participants were randomly assigned to complete a symptom assessment using either Ada or Rheport. The primary outcome was the correct identification of IRDs by the DDSSs, defined as the presence of any IRD in the list of suggested diagnoses by Ada or achieving a prespecified threshold score with Rheport. The gold standard was the diagnosis made by rheumatologists.ResultsA total of 600 patients were included, among whom 214 (35.7%) were diagnosed with an IRD. Most frequent IRD was rheumatoid arthritis with 69 (11.5%) patients. Rheport's disease suggestion and Ada's top 1 (D1) and top 5 (D5) disease suggestions demonstrated overall diagnostic accuracies of 52%, 63%, and 58%, respectively, for IRDs. Rheport showed a sensitivity of 62% and a specificity of 47% for IRDs. Ada's D1 and D5 disease suggestions showed a sensitivity of 52% and 66%, respectively, and a specificity of 68% and 54%, respectively, concerning IRDs. Ada's diagnostic accuracy regarding individual diagnoses was heterogenous, and Ada performed considerably better in identifying rheumatoid arthritis in comparison to other diagnoses (D1: 42%; D5: 64%). The Cohen κ statistic of Rheport for agreement on any rheumatic disease diagnosis with Ada D1 was 0.15 (95% CI 0.08-0.18) and with Ada D5 was 0.08 (95% CI 0.00-0.16), indicating poor agreement for the presence of any rheumatic disease between the 2 DDSSs.ConclusionsTo our knowledge, this is the largest comparative DDSS trial with actual use of DDSSs by patients. The diagnostic accuracies of both DDSSs for IRDs were not promising in this high-prevalence patient population. DDSSs may lead to a misuse of scarce health care resources. Our results underscore the need for stringent regulation and drastic improvements to ensure the safety and efficacy of DDSSs.Trial registrationGerman Register of Clinical Trials DRKS00017642; https://drks.de/search/en/trial/DRKS00017642.
| S-EPMC11303907 | biostudies-literature
Project description:BackgroundComputerized algorithms known as symptom checkers aim to help patients decide what to do should they have a new medical concern. However, despite widespread implementation, most studies on symptom checkers have involved simulated patients. Only limited evidence currently exists about symptom checker safety or accuracy when used by real patients. We developed a new prototype symptom checker and assessed its safety and accuracy in a prospective cohort of patients presenting to primary care and emergency departments with new medical concerns.MethodA prospective cohort study was done to assess the prototype's performance. The cohort consisted of adult patients (≥16 years old) who presented to hospital emergency departments and family physician clinics. Primary outcomes were safety and accuracy of triage recommendations to seek hospital care, seek primary care, or manage symptoms at home.ResultsData from 281 hospital patients and 300 clinic patients were collected and analyzed. Sensitivity to emergencies was 100% (10/10 encounters). Sensitivity to urgencies was 90% (73/81) and 97% (34/35) for hospital and primary care patients, respectively. The prototype was significantly more accurate than patients at triage (73% versus 58%, p<0.01). Compliance with triage recommendations in this cohort using this iteration of the symptom checker would have reduced hospital visits by 55% but cause potential harm in 2-3% from delay in care.InterpretationThe prototype symptom checker was superior to patients in deciding the most appropriate treatment setting for medical issues. This symptom checker could reduce a significant number of unnecessary hospital visits, with accuracy and safety outcomes comparable to existing data on telephone triage.
| S-EPMC8635379 | biostudies-literature
Project description:It is currently unknown which attitude-based profiles are associated with symptom checker use for self-triage. We sought to identify, among university students, attitude-based latent classes (population profiles) and the association between latent classes with the future use of symptom checkers for self-triage. Informed by the Technology Acceptance Model and a larger mixed methods study, a cross-sectional survey was developed and administered to students (aged between 18 and 34 years of age) at a University in Ontario. Latent class analysis (LCA) was used to identify attitude-based profiles that exist among the sample while general linear modeling was applied to identify the association between latent classes and future symptom checker use for self-triage. Of the 1,547 students who opened the survey link, 1,365 did not use a symptom checker in the past year and were thus identified as “non-users”. After removing missing data (remaining sample = n = 1,305), LCA revealed five attitude-based profiles: tech acceptors, tech rejectors, skeptics, tech seekers, and unsure acceptors. Tech acceptors and tech rejectors were the most and least prevalent classes, respectively. As compared to tech rejectors, tech seekers and unsure acceptors were the latent classes with the highest and lowest odds of future symptom checker use, respectively. After controlling for confounders, the effect of latent classes on symptom checker use remains significant (p-value < .0001) with the odds of future use in tech acceptors being 5.6 times higher than the odds of future symptom checker use in tech rejectors [CI: (3.458, 9.078); p-value < .0001]. Attitudes towards AI and symptom checker functionality result in different population profiles that have different odds of using symptom checkers for self-triage. Identifying a person’s or group’s membership to a population profile could help in developing and delivering tailored interventions aimed at maximizing use of validated symptom checkers.
| S-EPMC8565791 | biostudies-literature