Dataset Information

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study.

ABSTRACT:

Background

Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)-based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials.

Objective

The main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion.

Methods

This mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs.

Results

A total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario (P=.84), clinical questions (P=.99), and expert opinion (P=.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content.

Conclusions

This study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.

SUBMITTER: Hudon A

PROVIDER: S-EPMC11007379 | biostudies-literature | 2024 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study.

Hudon Alexandre A Kiepura Barnabé B Pelletier Myriam M Phan Véronique V

JMIR medical education 20240404

<h4>Background</h4>Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial int ...[more]

PMID: 38596832

Similar Datasets

Project description:BackgroundTelemedicine has grown exponentially in the United States over the past few decades, and contemporary trends in the health care environment are serving to fuel this growth into the future. Therefore, medical schools are learning to incorporate telemedicine competencies into the undergraduate medical education of future physicians so that they can more effectively leverage telemedicine technologies for improving the quality of care, increasing patient access, and reducing health care expense. This review articulates the efforts of allopathic-degree-granting medical schools in the United States to characterize and systematize the learnings that have been generated thus far in the domain of telemedicine training in undergraduate medical education.ObjectiveThe aim of this review was to collect and outline the current experiences and learnings that have been generated as medical schools have sought to implement telemedicine capacity-building into undergraduate medical education.MethodsWe performed a mixed-methods review, starting with a literature review via Scopus, tracking with Excel, and an email outreach effort utilizing telemedicine curriculum data gathered by the Liaison Committee on Medical Education. This outreach included 70 institutions and yielded 7 interviews, 4 peer-reviewed research papers, 6 online documents, and 3 completed survey responses.ResultsThere is an emerging, rich international body of learning being generated in the field of telemedicine training in undergraduate medical education. The integration of telemedicine-based lessons, ethics case-studies, clinical rotations, and even teleassessments are being found to offer great value for medical schools and their students. Most medical students find such training to be a valuable component of their preclinical and clinical education for a variety of reasons, which include fostering greater familiarity with telemedicine and increased comfort with applying telemedical approaches in their future careers.ConclusionsThese competencies are increasingly important in tackling the challenges facing health care in the 21st century, and further implementation of telemedicine curricula into undergraduate medical education is highly merited.

Project description:BackgroundThe integration of artificial intelligence (AI) technologies, such as ChatGPT, in the educational landscape has the potential to enhance the learning experience of medical informatics students and prepare them for using AI in professional settings. The incorporation of AI in classes aims to develop critical thinking by encouraging students to interact with ChatGPT and critically analyze the responses generated by the chatbot. This approach also helps students develop important skills in the field of biomedical and health informatics to enhance their interaction with AI tools.ObjectiveThe aim of the study is to explore the perceptions of students regarding the use of ChatGPT as a learning tool in their educational context and provide professors with examples of prompts for incorporating ChatGPT into their teaching and learning activities, thereby enhancing the educational experience for students in medical informatics courses.MethodsThis study used a mixed methods approach to gain insights from students regarding the use of ChatGPT in education. To accomplish this, a structured questionnaire was applied to evaluate students' familiarity with ChatGPT, gauge their perceptions of its use, and understand their attitudes toward its use in academic and learning tasks. Learning outcomes of 2 courses were analyzed to propose ChatGPT's incorporation in master's programs in medicine and medical informatics.ResultsThe majority of students expressed satisfaction with the use of ChatGPT in education, finding it beneficial for various purposes, including generating academic content, brainstorming ideas, and rewriting text. While some participants raised concerns about potential biases and the need for informed use, the overall perception was positive. Additionally, the study proposed integrating ChatGPT into 2 specific courses in the master's programs in medicine and medical informatics. The incorporation of ChatGPT was envisioned to enhance student learning experiences and assist in project planning, programming code generation, examination preparation, workflow exploration, and technical interview preparation, thus advancing medical informatics education. In medical teaching, it will be used as an assistant for simplifying the explanation of concepts and solving complex problems, as well as for generating clinical narratives and patient simulators.ConclusionsThe study's valuable insights into medical faculty students' perspectives and integration proposals for ChatGPT serve as an informative guide for professors aiming to enhance medical informatics education. The research delves into the potential of ChatGPT, emphasizes the necessity of collaboration in academic environments, identifies subject areas with discernible benefits, and underscores its transformative role in fostering innovative and engaging learning experiences. The envisaged proposals hold promise in empowering future health care professionals to work in the rapidly evolving era of digital health care.

Project description:BackgroundThere is an increasing demand for universal, high-quality access to palliative care in Austria. To ensure this, the implementation of palliative care in the medical studies curriculum is essential. This is the first study to investigate the state of undergraduate palliative care education at Austrian medical schools.MethodsFor this mixed-methods study with concurrent embedded design, expert interviews and online surveys were conducted between March and August 2022. The interviews were subjected to a thematic analysis according to Braun and Clarke, while the questionnaires were analysed descriptively-statistically. For the final integration, the results of both methods for each topic are presented and discussed complementarily. Both the primary qualitative and supportive quantitative data were collected to combine the advantages of the in-depth nature of the qualitative data and the consistent structure of the quantitative data to provide a more precise representation of the state of teaching.ResultsTwenty-two persons participated in the study, of whom twenty-one participated in the interview and eight in the questionnaire. The participants were experts in palliative care teaching at Austrian medical schools. Currently, palliative care is taught at seven out of the eight universities. Large differences were found in the number of hours, organisation, teaching formats, and interprofessional education. At present, three universities have a chair for palliative care and at least five universities have access to a palliative care unit.ConclusionUndergraduate palliative care education in Austria is very heterogeneous and does not meet the minimum standards suggested by the European Association for Palliative Care (EAPC) curriculum recommendations. However, several universities are planning measures to expand palliative care teaching, such as the introduction of mandatory teaching or the establishment of new teaching formats. Better coordination and networking within and between universities would be beneficial for the expansion and quality of teaching.

Project description:BackgroundThe purpose of this study was to evaluate the extent to which a longitudinal student-run clinic (SRC) is meeting its stated learning objectives, including providing critical community services and developing physicians who more fully appreciate the social factors affecting their patients' health.MethodsThis was a mixed methods program evaluation of an SRC at Mayo Clinic Alix School of Medicine (MCASOM). A survey was conducted of medical students who had participated in the clinic and seven interviews and three focus groups were conducted with SRC patients, students, faculty, staff, and board members. Transcripts were coded for systematic themes and sub-themes. Major themes were reported. Survey and interview data were integrated by comparing findings and discussing areas of convergence or divergence in order to more fully understand program success and potential areas for improvement.ResultsGreater than 85% of student survey respondents (N = 90) agreed or strongly agreed that the SRC met each of its objectives: to provide a vital community service, to explore social determinants of health (SDH), to understand barriers to healthcare access and to practice patience-centered examination. Qualitative data revealed that the SRC contextualized authentic patient care experiences early in students' medical school careers, but the depth of learning was variable between students. Furthermore, exposure to SDH through the program did not necessarily translate to student understanding of the impact of these social factors on patient's health nor did it clearly influence students' future practice goals.ConclusionsThe MCASOM SRC experience met core learning objectives, but opportunities to improve long-term impact on students were identified. Participation in the SRC enabled students to engage in patient care early in training that is representative of future practices. SRCs are an avenue by which students can gain exposure to real-world applications of SDH and barriers to healthcare access, but additional focus on faculty development and intentional reflection may be needed to translate this exposure to actionable student understanding of social factors that impact patient care.

Project description:BackgroundEmerging artificial intelligence (AI) technologies have diverse applications in medicine. As AI tools advance towards clinical implementation, skills in how to use and interpret AI in a healthcare setting could become integral for physicians. This study examines undergraduate medical students' perceptions of AI, educational opportunities about of AI in medicine, and the desired medium for AI curriculum delivery.MethodsA 32 question survey for undergraduate medical students was distributed from May-October 2021 to students to all 17 Canadian medical schools. The survey assessed the currently available learning opportunities about AI, the perceived need for learning opportunities about AI, and barriers to educating about AI in medicine. Interviews were conducted with participants to provide narrative context to survey responses. Likert scale survey questions were scored from 1 (disagree) to 5 (agree). Interview transcripts were analyzed using qualitative thematic analysis.ResultsWe received 486 responses from 17 of 17 medical schools (roughly 5% of Canadian undergraduate medical students). The mean age of respondents was 25.34, with 45% being in their first year of medical school, 27% in their 2nd year, 15% in their 3rd year, and 10% in their 4th year. Respondents agreed that AI applications in medicine would become common in the future (94% agree) and would improve medicine (84% agree Further, respondents agreed that they would need to use and understand AI during their medical careers (73% agree; 68% agree), and that AI should be formally taught in medical education (67% agree). In contrast, a significant number of participants indicated that they did not have any formal educational opportunities about AI (85% disagree) and that AI-related learning opportunities were inadequate (74% disagree). Interviews with 18 students were conducted. Emerging themes from the interviews were a lack of formal education opportunities and non-AI content taking priority in the curriculum.ConclusionA lack of educational opportunities about AI in medicine were identified across Canada in the participating students. As AI tools are currently progressing towards clinical implementation and there is currently a lack of educational opportunities about AI in medicine, AI should be considered for inclusion in formal medical curriculum.

Project description:IntroductionAs information and communication technology continues to shape the healthcare landscape, future medical practitioners need to be equipped with skills and competencies that ensure safe, high-quality, and person-centred healthcare in a digitised healthcare system. This study investigated undergraduate medical students' and medical educators' opinions of teleconsultation practice in general and their opinions of teleconsultation education.MethodsThis study used a cross-sectional, mixed-methods approach, utilising the additional coverage design to sequence and integrate qualitative and quantitative data. An online questionnaire was sent out to all medical schools in the UK, inviting undergraduate medical students and medical educators to participate. Questionnaire participants were given the opportunity to take part in a qualitative semi-structured interview. Descriptive and correlation analyses and a thematic analysis were conducted.ResultsA total of 248 participants completed the questionnaire and 23 interviews were conducted. Saving time and the reduced risks of transmitting infectious diseases were identified as common advantages of using teleconsultation. However, concerns about confidentiality and accessibility to services were expressed by students and educators. Eight themes were identified from the thematic analysis. The themes relevant to teleconsultation practice were (1) The benefit of teleconsultations, (2) A second-best option, (3) Patient choice, (4) Teleconsultations differ from in-person interactions, and (5) Impact on the healthcare system. The themes relevant to teleconsultation education were (6) Considerations and reflections on required skills, (7) Learning and teaching content, and (8) The future of teleconsultation education.DiscussionThe results of this study have implications for both medical practice and education. Patient confidentiality, safety, respecting patients' preferences, and accessibility are important considerations for implementing teleconsultations in practice. Education should focus on assessing the appropriateness of teleconsultations, offering accessible and equal care, and developing skills for effective communication and clinical reasoning. High-quality teleconsultation education can influence teleconsultation practice.

Dataset Information

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study.

Background

Objective

Methods

Results

Conclusions

Publications

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets