Dataset Information

ChatGPT and large language models (LLMs) awareness and use. A prospective cross-sectional survey of U.S. medical students.

ABSTRACT: Generative-AI (GAI) models like ChatGPT are becoming widely discussed and utilized tools in medical education. For example, it can be used to assist with studying for exams, shown capable of passing the USMLE board exams. However, there have been concerns expressed regarding its fair and ethical use. We designed an electronic survey for students across North American medical colleges to gauge their views on and current use of ChatGPT and similar technologies in May, 2023. Overall, 415 students from at least 28 medical schools completed the questionnaire and 96% of respondents had heard of ChatGPT and 52% had used it for medical school coursework. The most common use in pre-clerkship and clerkship phase was asking for explanations of medical concepts and assisting with diagnosis/treatment plans, respectively. The most common use in academic research was for proof reading and grammar edits. Respondents recognized the potential limitations of ChatGPT, including inaccurate responses, patient privacy, and plagiarism. Students recognized the importance of regulations to ensure proper use of this novel technology. Understanding the views of students is essential to crafting workable instructional courses, guidelines, and regulations that ensure the safe, productive use of generative-AI in medical school.

SUBMITTER: Ganjavi C

PROVIDER: S-EPMC11376538 | biostudies-literature | 2024 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

ChatGPT and large language models (LLMs) awareness and use. A prospective cross-sectional survey of U.S. medical students.

Ganjavi Conner C Eppler Michael M O'Brien Devon D Ramacciotti Lorenzo Storino LS Ghauri Muhammad Shabbeer MS Anderson Issac I Choi Jae J Dwyer Darby D Stephens Claudia C Shi Victoria V Ebert Madeline M Derby Michaela M Yazdi Bayan B Cacciamani Giovanni E GE

PLOS digital health 20240905 9

Generative-AI (GAI) models like ChatGPT are becoming widely discussed and utilized tools in medical education. For example, it can be used to assist with studying for exams, shown capable of passing the USMLE board exams. However, there have been concerns expressed regarding its fair and ethical use. We designed an electronic survey for students across North American medical colleges to gauge their views on and current use of ChatGPT and similar technologies in May, 2023. Overall, 415 students f ...[more]

PMID: 39236008

Similar Datasets

Project description:ObjectiveTo investigate the level of understanding and trust of medical students towards ChatGPT-like large language models, as well as their utilization and attitudes towards these models.MethodsData collection was concentrated from December 2023 to mid-January 2024, utilizing a self-designed questionnaire to assess the use of large language models among undergraduate medical students at Anhui Medical University. The normality of the data was confirmed with Shapiro-Wilk tests. We used Chi-square tests for comparisons of categorical variables, Mann-Whitney U tests for comparisons of ordinal variables and non-normal continuous variables between two groups, Kruskall-Wallis H tests for comparisons of ordinal variables between multiple groups, and Bonferroni tests for post hoc comparisons.ResultsA total of 1774 questionnaires were distributed and 1718 valid questionnaires were collected, with an effective rate of 96.84%. Among these students, 34.5% had heard and used large language models. There were statistically significant differences in the understanding of large language models between genders (p < 0.001), grade levels (junior-level students and senior-level students) (p = 0.03), and major (p < 0.001). Male, junior-level students, and public health management had a higher level of understanding of these models. Genders and majors had statistically significant effects on the degree of trust in large language models (p = 0.004; p = 0.02). Male and nursing students exhibited a higher degree of trust in large language models. As for usage, Male and junior-level students showed a significantly higher proportion of using these models for assisted learning (p < 0.001). Neutral sentiments were held by over two-thirds of the students (66.7%) regarding large language models, with only 51(3.0%) expressing pessimism. There were significant gender-based disparities in attitudes towards large language models, and male exhibited a more optimistic attitude towards these models (p < 0.001). Notably, among students with different levels of knowledge and trust in large language models, statistically significant differences were observed in their perceptions of the shortcomings and benefits of these models.ConclusionOur study identified gender, grade levels, and major as influential factors in students' understanding and utilization of large language models. This also suggested the feasibility of integrating large language models with traditional medical education to further enhance teaching effectiveness in the future.

Project description:BackgroundFeature selection is a critical step for translating advances afforded by systems-scale molecular profiling into actionable clinical insights. While data-driven methods are commonly utilized for selecting candidate genes, knowledge-driven methods must contend with the challenge of efficiently sifting through extensive volumes of biomedical information. This work aimed to assess the utility of large language models (LLMs) for knowledge-driven gene prioritization and selection.MethodsIn this proof of concept, we focused on 11 blood transcriptional modules associated with an Erythroid cells signature. We evaluated four leading LLMs across multiple tasks. Next, we established a workflow leveraging LLMs. The steps consisted of: (1) Selecting one of the 11 modules; (2) Identifying functional convergences among constituent genes using the LLMs; (3) Scoring candidate genes across six criteria capturing the gene's biological and clinical relevance; (4) Prioritizing candidate genes and summarizing justifications; (5) Fact-checking justifications and identifying supporting references; (6) Selecting a top candidate gene based on validated scoring justifications; and (7) Factoring in transcriptome profiling data to finalize the selection of the top candidate gene.ResultsOf the four LLMs evaluated, OpenAI's GPT-4 and Anthropic's Claude demonstrated the best performance and were chosen for the implementation of the candidate gene prioritization and selection workflow. This workflow was run in parallel for each of the 11 erythroid cell modules by participants in a data mining workshop. Module M9.2 served as an illustrative use case. The 30 candidate genes forming this module were assessed, and the top five scoring genes were identified as BCL2L1, ALAS2, SLC4A1, CA1, and FECH. Researchers carefully fact-checked the summarized scoring justifications, after which the LLMs were prompted to select a top candidate based on this information. GPT-4 initially chose BCL2L1, while Claude selected ALAS2. When transcriptional profiling data from three reference datasets were provided for additional context, GPT-4 revised its initial choice to ALAS2, whereas Claude reaffirmed its original selection for this module.ConclusionsTaken together, our findings highlight the ability of LLMs to prioritize candidate genes with minimal human intervention. This suggests the potential of this technology to boost productivity, especially for tasks that require leveraging extensive biomedical knowledge.

Project description:BackgroundPhysicians often report low levels of confidence in diagnosing and treating female sexual dysfunction (FSD), which may stem from inadequate education and exposure to the topic.AimThe study sought to determine the extent to which FSD is covered in undergraduate medical education and evaluate student comfort with the topic and familiarity with treatments.MethodsWe created a novel, 50-question survey to be administered online to any current U.S. medical student. Obstetrician-gynecologist clerkship directors at 146 U.S. medical schools were contacted and asked to invite any current student at their school to participate. A link to the electronic REDCap (Research Electronic Data Capture) survey was distributed to eligible students via the clerkship directors. The survey collected data regarding (1) coverage of FSD and male sexual dysfunction (MSD) in preclinical and clinical curricula, (2) student self-ratings of comfort in hypothetical scenarios in which a patient exhibits symptoms of FSD or MSD, and (3) student familiarity with treatments for FSD and MSD.OutcomesOutcomes included the proportion of students reporting that their school covered FSD/MSD in its preclinical/clinical curriculum, the mean comfort ratings for each of the FSD and MSD scenarios, and the proportion of students indicating knowledge of various FSD and MSD treatments.ResultsA smaller proportion of students (N = 236) reported receiving instruction in FSD (58.5%) compared with MSD (78.4%) in their preclinical curriculum (P < .001). Students' average self-ratings of comfort in the sexual dysfunction scenarios were significantly lower for patients with symptoms of FSD compared with MSD (P < .001). Students had higher average self-ratings of confidence in FSD scenarios if their intended specialty was obstetrician-gynecologist (P = .003), if their school included FSD in its clinical curriculum (P = .01), and if they had ever participated in the care of a patient with FSD (P = .006).Clinical implicationsThere are important gaps in the coverage of FSD in undergraduate medical education that may be mitigated through improvements to curriculum and increased exposure to patients with FSD.Strengths and limitationsThis is the first study, to our knowledge, to directly survey medical students regarding their educational experience and comfort with FSD. Our study was limited by a small sample size, the use of a novel and nonvalidated questionnaire, and the potential for bias given our sampling method.ConclusionMedical schools must work toward improving instruction in FSD for their students to address these disparities and improve students' comfort with the topic.

Project description:PurposeLarge language models (LLMs) are deep learning models designed to comprehend and generate meaningful responses, which have gained public attention in recent years. The purpose of this study is to evaluate and compare the performance of LLMs in answering questions regarding breast cancer in the Chinese context.Material and methodsChatGPT, ERNIE Bot, and ChatGLM were chosen to answer 60 questions related to breast cancer posed by two oncologists. Responses were scored as comprehensive, correct but inadequate, mixed with correct and incorrect data, completely incorrect, or unanswered. The accuracy, length, and readability among answers from different models were evaluated using statistical software.ResultsChatGPT answered 60 questions, with 40 (66.7%) comprehensive answers and six (10.0%) correct but inadequate answers. ERNIE Bot answered 60 questions, with 34 (56.7%) comprehensive answers and seven (11.7%) correct but inadequate answers. ChatGLM generated 60 answers, with 35 (58.3%) comprehensive answers and six (10.0%) correct but inadequate answers. The differences for chosen accuracy metrics among the three LLMs did not reach statistical significance, but only ChatGPT demonstrated a sense of human compassion. The accuracy of the three models in answering questions regarding breast cancer treatment was the lowest, with an average of 44.4%. ERNIE Bot's responses were significantly shorter compared to ChatGPT and ChatGLM (p < .001 for both). The readability scores of the three models showed no statistical significance.ConclusionsIn the Chinese context, the capabilities of ChatGPT, ERNIE Bot, and ChatGLM are similar in answering breast cancer-related questions at present. These three LLMs may serve as adjunct informational tools for breast cancer patients in the Chinese context, offering guidance for general inquiries. However, for highly specialized issues, particularly in the realm of breast cancer treatment, LLMs cannot deliver reliable performance. It is necessary to utilize them under the supervision of healthcare professionals.

Dataset Information

ChatGPT and large language models (LLMs) awareness and use. A prospective cross-sectional survey of U.S. medical students.

Publications

ChatGPT and large language models (LLMs) awareness and use. A prospective cross-sectional survey of U.S. medical students.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets