Dataset Information

Using longitudinal progress test data to determine the effect size of learning in undergraduate medical education - a retrospective, single-center, mixed model analysis of progress testing results.

ABSTRACT: Medical education research focuses on the development of efficient learning methods promoting the acquisition of student's knowledge and competencies. Evaluation of any modification of educational approaches needs to be evaluated accordingly and a reliable effect size needs to be reached. Our aim is to provide a methodological basis to calculate effect sizes from longitudinal progress test data that can be used as reference values in further research. We used longitudinally collected progress test data and evaluated the increasing knowledge of medical students from the first to the fifth academic year. Students were asked to participate in the progress test, which consists of 200 multiple-choice questions in single best answer format with an additional 'don't know' option. All available individual test scores of all progress tests (n = 10) administered between April 2012 and October 2017 were analyzed. Due to the large amount of missing test results, e.g., from students at the beginning of their studies, a linear mixed model was fitted to include all collected data. In total, we analyzed 6324 test scores provided by 2587 medical students. Mean score for medical knowledge (% correct answers) increases from 16.6% (SD: 10.8%) to 51.0% (SD: 15.7%, overall effects size using linear mixed models d = 1.55). Medical students showed a learning effect of d = 0.54 (total gain: 6.9%) between the 1^st and 2^nd, d = 0.88 (total gain: 12.0%) between the 2^nd and 3^rd, d = 0.60 (total gain: 7.9%) between the 3^rd and 4^th and d = 0.58 (total gain: 7.9%) between the 4^th and 5^th study year. We demonstrated that incomplete data from longitudinally collected progress tests can be used to acquire reliable effect size estimates. The demonstrated effects size between d = 0.53-0.9 by study year may help researchers to design studies in medical education.

SUBMITTER: Gorlich D

PROVIDER: S-EPMC8409971 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:Telemedicine has grown exponentially in the United States over the past few decades, and contemporary trends in the health care environment are serving to fuel this growth into the future. Therefore, medical schools are learning to incorporate telemedicine competencies into the undergraduate medical education of future physicians so that they can more effectively leverage telemedicine technologies for improving the quality of care, increasing patient access, and reducing health care expense. This review articulates the efforts of allopathic-degree-granting medical schools in the United States to characterize and systematize the learnings that have been generated thus far in the domain of telemedicine training in undergraduate medical education. OBJECTIVE:The aim of this review was to collect and outline the current experiences and learnings that have been generated as medical schools have sought to implement telemedicine capacity-building into undergraduate medical education. METHODS:We performed a mixed-methods review, starting with a literature review via Scopus, tracking with Excel, and an email outreach effort utilizing telemedicine curriculum data gathered by the Liaison Committee on Medical Education. This outreach included 70 institutions and yielded 7 interviews, 4 peer-reviewed research papers, 6 online documents, and 3 completed survey responses. RESULTS:There is an emerging, rich international body of learning being generated in the field of telemedicine training in undergraduate medical education. The integration of telemedicine-based lessons, ethics case-studies, clinical rotations, and even teleassessments are being found to offer great value for medical schools and their students. Most medical students find such training to be a valuable component of their preclinical and clinical education for a variety of reasons, which include fostering greater familiarity with telemedicine and increased comfort with applying telemedical approaches in their future careers. CONCLUSIONS:These competencies are increasingly important in tackling the challenges facing health care in the 21st century, and further implementation of telemedicine curricula into undergraduate medical education is highly merited.

Project description:BackgroundThere is an increasing demand for universal, high-quality access to palliative care in Austria. To ensure this, the implementation of palliative care in the medical studies curriculum is essential. This is the first study to investigate the state of undergraduate palliative care education at Austrian medical schools.MethodsFor this mixed-methods study with concurrent embedded design, expert interviews and online surveys were conducted between March and August 2022. The interviews were subjected to a thematic analysis according to Braun and Clarke, while the questionnaires were analysed descriptively-statistically. For the final integration, the results of both methods for each topic are presented and discussed complementarily. Both the primary qualitative and supportive quantitative data were collected to combine the advantages of the in-depth nature of the qualitative data and the consistent structure of the quantitative data to provide a more precise representation of the state of teaching.ResultsTwenty-two persons participated in the study, of whom twenty-one participated in the interview and eight in the questionnaire. The participants were experts in palliative care teaching at Austrian medical schools. Currently, palliative care is taught at seven out of the eight universities. Large differences were found in the number of hours, organisation, teaching formats, and interprofessional education. At present, three universities have a chair for palliative care and at least five universities have access to a palliative care unit.ConclusionUndergraduate palliative care education in Austria is very heterogeneous and does not meet the minimum standards suggested by the European Association for Palliative Care (EAPC) curriculum recommendations. However, several universities are planning measures to expand palliative care teaching, such as the introduction of mandatory teaching or the establishment of new teaching formats. Better coordination and networking within and between universities would be beneficial for the expansion and quality of teaching.

Project description:BackgroundUndergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)-based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials.ObjectiveThe main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion.MethodsThis mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs.ResultsA total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario (P=.84), clinical questions (P=.99), and expert opinion (P=.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content.ConclusionsThis study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.

Project description:BackgroundThe purpose of this study was to evaluate the extent to which a longitudinal student-run clinic (SRC) is meeting its stated learning objectives, including providing critical community services and developing physicians who more fully appreciate the social factors affecting their patients' health.MethodsThis was a mixed methods program evaluation of an SRC at Mayo Clinic Alix School of Medicine (MCASOM). A survey was conducted of medical students who had participated in the clinic and seven interviews and three focus groups were conducted with SRC patients, students, faculty, staff, and board members. Transcripts were coded for systematic themes and sub-themes. Major themes were reported. Survey and interview data were integrated by comparing findings and discussing areas of convergence or divergence in order to more fully understand program success and potential areas for improvement.ResultsGreater than 85% of student survey respondents (N = 90) agreed or strongly agreed that the SRC met each of its objectives: to provide a vital community service, to explore social determinants of health (SDH), to understand barriers to healthcare access and to practice patience-centered examination. Qualitative data revealed that the SRC contextualized authentic patient care experiences early in students' medical school careers, but the depth of learning was variable between students. Furthermore, exposure to SDH through the program did not necessarily translate to student understanding of the impact of these social factors on patient's health nor did it clearly influence students' future practice goals.ConclusionsThe MCASOM SRC experience met core learning objectives, but opportunities to improve long-term impact on students were identified. Participation in the SRC enabled students to engage in patient care early in training that is representative of future practices. SRCs are an avenue by which students can gain exposure to real-world applications of SDH and barriers to healthcare access, but additional focus on faculty development and intentional reflection may be needed to translate this exposure to actionable student understanding of social factors that impact patient care.

Project description:BackgroundThe UMAT is widely used for selection into undergraduate medical and dental courses in Australia and New Zealand (NZ). It tests aptitudes thought to be especially relevant to medical studies and consists of 3 sections - logical reasoning and problem solving (UMAT-1), understanding people (UMAT-2) and non-verbal reasoning (UMAT-3). A substantial proportion of all candidates re-sit the UMAT. Re-sitting raises the issue as to what might be the precise magnitude and determinants of any practice effects on the UMAT and their implications for equity in subsequent selection processes.MethodsBetween 2000 and 2012, 158,909 UMAT assessments were completed. From these, 135,833 cases were identified where a candidate had sat once or more during that period with 117,505 cases (86.5%) having sat once, 14,739 having sat twice (10.9%), 2,752 thrice (2%) and 837, 4 or more times (0.6%). Subsequent analyses determined predictors of multiple re-sits as well as the magnitude and socio-demographic determinants of any practice effects.ResultsIncreased likelihood of re-sitting the UMAT twice or more was predicted by being male, of younger age, being from a non-English language speaking background and being from NZ and for Australian candidates, being urban rather than rurally based. For those who sat at least twice, the total UMAT score between a first and second attempt improved by 10.7 ± 0.2 percentiles, UMAT-1 by 8.3 ± 0.2 percentiles, UMAT-2 by 8.3 ± 0.2 percentiles and UMAT-3 by 7.7 ± 0.2 percentiles. An increase in total UMAT percentile score on re-testing was predicted by a lower initial score and being a candidate from NZ rather than from Australia while a decrease was related to increased length of time since initially sitting the test, older age and non-English language background.ConclusionsRe-sitting the UMAT augments performance in each of its components together with the total UMAT percentile score. Whether this increase represents just an improvement in performance or an improvement in understanding of the variables and therefore competence needs to be further defined. If only the former, then practice effects may be introducing inequity in student selection for medical or dental schools in Australia or NZ.

Dataset Information

Using longitudinal progress test data to determine the effect size of learning in undergraduate medical education - a retrospective, single-center, mixed model analysis of progress testing results.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets