Dataset Information

Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study.

ABSTRACT: OBJECTIVES:The study aimed to compare candidate performance between traditional best-of-five single-best-answer (SBA) questions and very-short-answer (VSA) questions, in which candidates must generate their own answers of between one and five words. The primary objective was to determine if the mean positive cue rate for SBAs exceeded the null hypothesis guessing rate of 20%. DESIGN:This was a cross-sectional study undertaken in 2018. SETTING:20 medical schools in the UK. PARTICIPANTS:1417 volunteer medical students preparing for their final undergraduate medicine examinations (total eligible population across all UK medical schools approximately 7500). INTERVENTIONS:Students completed a 50-question VSA test, followed immediately by the same test in SBA format, using a novel digital exam delivery platform which also facilitated rapid marking of VSAs. MAIN OUTCOME MEASURES:The main outcome measure was the mean positive cue rate across SBAs: the percentage of students getting the SBA format of the question correct after getting the VSA format incorrect. Internal consistency, item discrimination and the pass rate using Cohen standard setting for VSAs and SBAs were also evaluated, and a cost analysis in terms of marking the VSA was performed. RESULTS:The study was completed by 1417 students. Mean student scores were 21 percentage points higher for SBAs. The mean positive cue rate was 42.7% (95% CI 36.8% to 48.6%), one-sample t-test against ?20%: t=7.53, p<0.001. Internal consistency was higher for VSAs than SBAs and the median item discrimination equivalent. The estimated marking cost was £2655 ($3500), with 24.5?hours of clinician time required (1.25?s per student per question). CONCLUSIONS:SBA questions can give a false impression of students' competence. VSAs appear to have greater authenticity and can provide useful information regarding students' cognitive errors, helping to improve learning as well as assessment. Electronic delivery and marking of VSAs is feasible and cost-effective.

SUBMITTER: Sam AH

PROVIDER: S-EPMC6773319 | biostudies-literature | 2019 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study.

Sam Amir H AH Westacott Rachel R Gurnell Mark M Wilson Rebecca R Meeran Karim K Brown Celia C

BMJ open 20190926 9

<h4>Objectives</h4>The study aimed to compare candidate performance between traditional best-of-five single-best-answer (SBA) questions and very-short-answer (VSA) questions, in which candidates must generate their own answers of between one and five words. The primary objective was to determine if the mean positive cue rate for SBAs exceeded the null hypothesis guessing rate of 20%.<h4>Design</h4>This was a cross-sectional study undertaken in 2018.<h4>Setting</h4>20 medical schools in the UK.<h ...[more]

PMID: 31558462

Similar Datasets

Project description:Multiple choice questions (MCQs) offer high reliability and easy machine-marking, but allow for cueing and stimulate recognition-based learning. Very short answer questions (VSAQs), which are open-ended questions requiring a very short answer, may circumvent these limitations. Although VSAQ use in medical assessment increases, almost all research on reliability and validity of VSAQs in medical education has been performed by a single research group with extensive experience in the development of VSAQs. Therefore, we aimed to validate previous findings about VSAQ reliability, discrimination, and acceptability in undergraduate medical students and teachers with limited experience in VSAQs development. To validate the results presented in previous studies, we partially replicated a previous study and extended results on student experiences. Dutch undergraduate medical students (n = 375) were randomized to VSAQs first and MCQs second or vice versa in a formative exam in two courses, to determine reliability, discrimination, and cueing. Acceptability for teachers (i.e., VSAQ review time) was determined in the summative exam. Reliability (Cronbach's α) was 0.74 for VSAQs and 0.57 for MCQs in one course. In the other course, Cronbach's α was 0.87 for VSAQs and 0.83 for MCQs. Discrimination (average Rir) was 0.27 vs. 0.17 and 0.43 vs. 0.39 for VSAQs vs. MCQs, respectively. Reviewing time of one VSAQ for the entire student cohort was ±2 minutes on average. Positive cueing occurred more in MCQs than in VSAQs (20% vs. 4% and 20.8% vs. 8.3% of questions per person in both courses). This study validates the positive results regarding VSAQs reliability, discrimination, and acceptability in undergraduate medical students. Furthermore, we demonstrate that VSAQ use is reliable among teachers with limited experience in writing and marking VSAQs. The short learning curve for teachers, favourable marking time and applicability regardless of the topic suggest that VSAQs might also be valuable beyond medical assessment.

Project description:During the preclinical years, single-best-answer multiple-choice questions (SBA-MCQs) are often used to test the higher-order cognitive processes of medical students (such as application and analysis) while simultaneously assessing lower-order processes (like knowledge and comprehension). Consequently, it can be difficult to pinpoint which learning outcome has been achieved or needs improvement. We developed a new scoring system for SBA-MCQs using a step-by-step methodology to evaluate each learning outcome independently. Enrolled in this study were third-year medical students (n = 316) who had registered in the basic microbiology course at the Faculty of Medicine, Siriraj Hospital, Mahidol University during the academic year 2017. A step-by-step SBA-MCQ with a new scoring system was created and used as a tool to evaluate the validity of the traditional SBA-MCQs that assess two separate outcomes simultaneously. The scores for the two methods, in percentages, were compared using two different questions (SBA-MCQ1 and SBA-MCQ2). SBA-MCQ1 tested the students' knowledge of the causative agent of a specific infectious disease and the basic characteristics of the microorganism, while SBA-MCQ2 tested their knowledge of the causative agent of a specific infectious disease and the pathogenic mechanism of the microorganism. The mean score obtained with the traditional SBA-MCQs was significantly lower than that obtained with the step-by-step SBA-MCQs (85.9% for the traditional approach versus 90.9% for step-by-step SBA-MCQ1; p < 0.001; and 81.5% for the traditional system versus 87.4% for step-by-step SBA-MCQ2; p < 0.001). Moreover, 65.8% and 87.8% of the students scored lower with the traditional SBA-MCQ1 and the traditional SBA-MCQ2, respectively, than with the corresponding sets of step-by-step SBA-MCQ questions. These results suggest that traditional SBA-MCQ scores need to be interpreted with caution because they have the potential to underestimate the learning achievement of students. Therefore, the step-by-step SBA-MCQ is preferable to the traditional SBA-MCQs and is recommended for use in examinations during the preclinical years.

Project description:BACKGROUND:Empathy is paramount in the doctor-patient relationship being a comprehensive and multidimensional concept. Self-ratings of empathy change across the years of medical education only when the Jefferson Scale of Physicians Empathy is used, with a worrying decrease being found throughout the years in Medicine Schools. As there are only few studies on the factors influencing medical student's empathy, particularly of the curricular model, this study aimed to compare the levels of empathy of medicine students of two Schools with different curricular models in central Portugal, the Faculty of Medicine - University of Coimbra (FMUC) and the Faculty of Health Sciences - University of Beira Interior (FCS-UBI). METHODS:Cross-sectional observational study with the Jefferson Scale of Physician Empathy - students' Portuguese version (JSPE - spv) to 1st, 3rd and 6th year students of the 2017/2018 academic year with descriptive and inferential statistical analysis (p?<?0.05). RESULTS:Size representative sample of 795 students. Higher total empathy score (TES) (p?=?0.008) and "Perspective taking" (p?=?0.001) in FCS-UBI were found. JSPE-TES was higher in FCS-UBI, 3rd year (p?=?0.038). Higher FCS-UBI "Perspective taking" in the 1st year (p?=?0.030) and 6th year (p?=?0.044), for "Compassionate care" in the 3rd (p?=?0.019) and for "Standing in the patient's shoes" in the 1st year (p?=?0.018) and in FMUC for "Compassionate care" in the 1st year (p?=?0.037) and the "Standing in the patient's shoes" in year 3 (p?=?0.002) were found. Higher levels of empathy were found in FCS-UBI female students, for JSPE-TES (p?=?0.045) and "Perspective taking" (p?=?0.001). CONCLUSION:Higher empathy levels in FCS-UBI were found, with different results in the third year suggesting influence of the medical course teaching characteristics between the two Medicine schools, student's empathy levels being higher when earlier and more intense contact with patients accompanied by skilled tutors was developed.

Dataset Information

Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study.

Publications

Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets