Dataset Information

Examining Real-World Medication Consultations and Drug-Herb Interactions: ChatGPT Performance Evaluation.

ABSTRACT:

Background

Since OpenAI released ChatGPT, with its strong capability in handling natural tasks and its user-friendly interface, it has garnered significant attention.

Objective

A prospective analysis is required to evaluate the accuracy and appropriateness of medication consultation responses generated by ChatGPT.

Methods

A prospective cross-sectional study was conducted by the pharmacy department of a medical center in Taiwan. The test data set comprised retrospective medication consultation questions collected from February 1, 2023, to February 28, 2023, along with common questions about drug-herb interactions. Two distinct sets of questions were tested: real-world medication consultation questions and common questions about interactions between traditional Chinese and Western medicines. We used the conventional double-review mechanism. The appropriateness of each response from ChatGPT was assessed by 2 experienced pharmacists. In the event of a discrepancy between the assessments, a third pharmacist stepped in to make the final decision.

Results

Of 293 real-world medication consultation questions, a random selection of 80 was used to evaluate ChatGPT's performance. ChatGPT exhibited a higher appropriateness rate in responding to public medication consultation questions compared to those asked by health care providers in a hospital setting (31/51, 61% vs 20/51, 39%; P=.01).

Conclusions

The findings from this study suggest that ChatGPT could potentially be used for answering basic medication consultation questions. Our analysis of the erroneous information allowed us to identify potential medical risks associated with certain questions; this problem deserves our close attention.

SUBMITTER: Hsu HY

PROVIDER: S-EPMC10477918 | biostudies-literature | 2023 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Examining Real-World Medication Consultations and Drug-Herb Interactions: ChatGPT Performance Evaluation.

Hsu Hsing-Yu HY Hsu Kai-Cheng KC Hou Shih-Yen SY Wu Ching-Lung CL Hsieh Yow-Wen YW Cheng Yih-Dih YD

JMIR medical education 20230821

<h4>Background</h4>Since OpenAI released ChatGPT, with its strong capability in handling natural tasks and its user-friendly interface, it has garnered significant attention.<h4>Objective</h4>A prospective analysis is required to evaluate the accuracy and appropriateness of medication consultation responses generated by ChatGPT.<h4>Methods</h4>A prospective cross-sectional study was conducted by the pharmacy department of a medical center in Taiwan. The test data set comprised retrospective medi ...[more]

PMID: 37561097

Similar Datasets

Project description:ObjectiveTo gather real-world evidence on antiseizure medications (ASMs) treatment patterns and related outcomes in patients with drug-resistant focal epilepsy.MethodsMedical insurance claims from the start of 2014 till the end of 2019 were used. Patient selection criteria included International Classification of Diseases (ICD) codes followed by documented ASM use. Baseline patient demographics along with ASM and rescue medication use patterns and related patient outcome were documented for first (index) ASM regimen. Patients who failed the first regimen and then failed the second regimen were considered drug resistant. Multivariate analyses were performed to identify risks and other characteristics for positive or negative treatment outcomes.ResultsStudy cohort consisted of 46 474 patients with a mean age of 47.23 (SD: 16.94). Levetiracetam was the most first-encountered ASM (37.94%). At baseline, 87.14% were treated with ASMs prior to having study-confirmed diagnoses. Mental comorbidities were present in 37.86% of patients. After first-year ASM treatment, 34.61% of patients persisted on their index regimen and 5.91% were seizure-free. Patients failing first ASM regimen numbered 12 868 (27.69%). Drug-resistant patients who failed first and then second ASM regimens numbered 6335 (49.23%). Percentages of patients who had successful second treatment and seizure-free were 21.32 and 3.65, respectively. Initiating patients on lamotrigine or carbamazepine (relative to levetiracetam), baseline use of index ASM, rescue medications, and older age or male gender all lowered the risk for treatment failure. Having higher comorbidity, comorbid mental illness, headache, or neoplasty increased such a risk. Baseline use of index ASM, depressive episode, or anxiety disorder all entailed higher risk of failing second ASM treatment.SignificanceOverall, reported findings indicated that patient history at baseline and the early selection of an ASM all influenced treatment outcomes. Findings pointed to the complex nature of ASM treatment in drug-resistant focal epilepsy patients calling for additional research to identify the optimal treatment to achieve beneficial patient outcomes.

Dataset Information

Examining Real-World Medication Consultations and Drug-Herb Interactions: ChatGPT Performance Evaluation.

Background

Objective

Methods

Results

Conclusions

Publications

Examining Real-World Medication Consultations and Drug-Herb Interactions: ChatGPT Performance Evaluation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets