Dataset Information

Agreement between medical students' peer assessments and faculty assessments in advanced resuscitation skills examinations in South Korea.

ABSTRACT:

Purpose

In medical education, peer assessment is considered to be an effective learning strategy. Although several studies have examined agreement between peer and faculty assessments regarding basic life support (BLS), few studies have done so for advanced resuscitation skills (ARS) such as intubation and defibrillation. Therefore, this study aimed to determine the degree of agreement between medical students' and faculty assessments of ARS examinations.

Methods

This retrospective explorative study was conducted during the emergency medicine (EM) clinical clerkship of fourth-year medical students from April to July 2020. A faculty assessor (FA) and a peer assessor (PA) assessed each examinee's resuscitation skills (including BLS, intubation, and defibrillation) using a checklist that consisted of 20 binary items (performed or not performed) and 1 global proficiency rating using a 5-point Likert scale. The prior examinee assessed the next examinee after feedback and training as a PA. All 54 students participated in peer assessment. The assessments of 44 FA/PA pairs were analyzed using the intraclass correlation coefficient (ICC) and Gwet's first-order agreement coefficient.

Results

The PA scores were higher than the FA scores (mean±standard deviation, 20.2±2.5 [FA] vs. 22.3±2.4 [PA]; P<0.001). The agreement was poor to moderate for the overall checklist (ICC, 0.55; 95% confidence interval [CI], 0.31 to 0.73; P<0.01), BLS (ICC, 0.19; 95% CI, -0.11 to 0.46; P<0.10), intubation (ICC, 0.51; 95% CI, 0.26 to 0.70; P<0.01), and defibrillation (ICC, 0.49; 95% CI, 0.23 to 0.68; P<0.01).

Conclusion

Senior medical students showed unreliable agreement in ARS assessments compared to faculty assessments. If a peer assessment is planned in skills education, comprehensive preparation and sufficient assessor training should be provided in advance.

SUBMITTER: Jeong J

PROVIDER: S-EPMC8089466 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundThe shift from a more didactic to student-centred pedagogical approach has led to the implementation of new information communication technology (ICT) innovations and curricula. Consequently, analysis of the digital competency of both faculty and students is of increasing importance. The aim of this research is to measure and compare the internet skills of medical school faculty and students and to investigate any potential skills gap between the two groups.MethodsA survey of medical school faculty and students across three universities in Ireland was carried out using a validated instrument (Internet Skills Scale) measuring five internet skills (Operational, Information Navigation, Social, Creative and Mobile). Three focus groups comprising a total of fifteen students and four semi-structured interviews with faculty across three institutions were carried out to explore further findings and perceptions towards digital literacy, give further insight and add context to the findings.ResultsSeventy-eight medical faculty (response rate 45%) and 401 students (response rate 15%) responded to the survey. Mean scores for each internet skill were high (above 4 out of 5) for all skills apart from Creative (mean of 3.08 for students and 3.10 for faculty). There were no large differences between student and faculty scores across the five skills. Qualitative results supported survey findings with a deeper investigation into topics such as online professionalism, use of licencing and mobile application development. Needs based skills training and support were highlighted as areas for faculty development.ConclusionBoth medical educators and students tend to have similar competencies with respect to internet skills. When implementing online and distance learning methodologies however, medical schools need to ensure appropriate skills training and support for faculty as well as providing targeted training to improve the creative skills of both their educators and students.

Project description:ObjectivesThe qualitative subjective assessment has been exercised either by self-reflection (self-assessment (SA)) or by an observer (peer assessment (PA)) and is considered to play an important role in students' development. The objectivity of PA and SA by students as well as those by faculty examiners has remained debated. This matters most when it comes to a high-stakes examination. We explored the degree of objectivity in PA, SA, as well as the global rating by examiners being Examiners' Subjective Assessment (ESA) compared with Objective Structured Clinical Examinations (OSCE).DesignProspective cohort study.SettingUndergraduate medical students at Alfaisal University, Riyadh.ParticipantsAll second-year medical students (n=164) of genders, taking a course to learn clinical history taking and general physical examination.Main outcome measuresA Likert scale questionnaire was distributed among the participants during selected clinical skills sessions. Each student was evaluated randomly by peers (PA) as well as by himself/herself (SA). Two OSCEs were conducted where students were assessed by an examiner objectively as well as subjectively (ESA) for a global rating of confidence and well-preparedness. OSCE-1 had fewer topics and stations, whereas OSCE-2 was terminal and full scale.ResultsOSCE-1 (B=0.10) and ESA (B=8.16) predicted OSCE-2 scores. 'No nervousness' in PA (r=0.185, p=0.018) and 'confidence' in SA (r=0.207, p=0.008) correlated with 'confidence' in ESA. In 'well-preparedness', SA correlated with ESA (r=0.234, p=0.003).ConclusionsOSCE-1 and ESA predicted students' performance in the OSCE-2, a high-stakes evaluation, indicating practical 'objectivity' in ESA, whereas SA and PA had minimal predictive role. Certain components of SA and PA correlated with ESA, suggesting partial objectivity given the limited objectiveness of ESA. Such difference in 'qualitative' objectivity probably reflects experience. Thus, subjective assessment can be used with some degree of objectivity for continuous assessment.

Project description:BackgroundTeaching and assessment of communication skills have become essential in medical education. The Objective Structured Clinical Examination (OSCE) has been found as an appropriate means to assess communication skills within medical education. Studies have demonstrated the importance of a valid assessment of medical students' communication skills. Yet, the validity of the performance scores depends fundamentally on the quality of the rating scales used in an OSCE. Thus, this systematic review aimed at providing an overview of existing rating scales, describing their underlying definition of communication skills, determining the methodological quality of psychometric studies and the quality of psychometric properties of the identified rating scales.MethodsWe conducted a systematic review to identify psychometrically tested rating scales, which have been applied in OSCE settings to assess communication skills of medical students. Our search strategy comprised three databases (EMBASE, PsycINFO, and PubMed), reference tracking and consultation of experts. We included studies that reported psychometric properties of communication skills assessment rating scales used in OSCEs by examiners only. The methodological quality of included studies was assessed using the COnsensus based Standards for the selection of health status Measurement INstruments (COSMIN) checklist. The quality of psychometric properties was evaluated using the quality criteria of Terwee and colleagues.ResultsData of twelve studies reporting on eight rating scales on communication skills assessment in OSCEs were included. Five of eight rating scales were explicitly developed based on a specific definition of communication skills. The methodological quality of studies was mainly poor. The psychometric quality of the eight rating scales was mainly intermediate.DiscussionOur results reveal that future psychometric evaluation studies focusing on improving the methodological quality are needed in order to yield psychometrically sound results of the OSCEs assessing communication skills. This is especially important given that most OSCE rating scales are used for summative assessment, and thus have an impact on medical students' academic success.

Project description:In the model medical curriculum HannibaL at Hannover Medical School (MHH, Hannover, Germany), communication skills in taking case histories and disclosing diagnoses (breaking bad news) are assessed through an objective structured clinical examination (OSCE). This is part of the examinations which at the MHH represent the equivalent to the First Part of the Medical Examinations. The second year doctor-patient communication course preparing for these examinations was evaluated during the 2009/10 academic year.Using questionnaires specific to the learning objectives, learning needs were assessed, pre-post comparisons of self-assessed competencies were performed and key teaching methods were evaluated (5-point Likert scales, "5"=fully agree). At T0 (start of the course) 267 students participated (response rate: 93.7%), of which 180 filled out the T1 questionnaire during the last session of the course (67.4%). Within-subject analyses of variance and paired t-tests were conducted.The highest learning needs were found for the "to show how"-items regarding history taking and disclosing diagnoses (M=4.4). The T1-T0 comparisons showed the greatest improvements for history taking ("to know how": mean difference = +1.7, "to show how": +1.8, p<.0001 as with all tests) and the "to know how"-item regarding the disclosure of diagnoses (+1.6), followed by the "to show how"-items on disclosing a diagnosis (+1.4), shared decision making (+1.2), self-assessing one's own strengths/weaknesses (+1.0) and confidently approaching new patients (+0.7). Students with T0 values of 1 or 2 on the respective scales improved on average by 2.2 points across all items, students with the value of 3 by 1.1, and from 4 or 5 by 0.1. Methodically, the use of simulated patients was rated the most helpful (M=4.8, 87% with the scale value 5). This doctor-patient communication course is associated with substantial improvements regarding all key learning objectives. Regarding methods, the deployed simulated patients (2-4 per 10-student-course group in 3 of the 7 course sessions, respectively) were rated the most helpful. The present evaluation calls for both further development of the doctor-patient communication curriculum at the MHH and joint activities across medical schools, which are discussed at the end of the paper.