Dataset Information

The Acute Dialysis Orders Objective Structured Clinical Examination (OSCE): Fellow Performance on a Formative Assessment of Acute Kidney Replacement Therapy Competence.

ABSTRACT:

Background and objectives

Acute kidney replacement therapy (KRT) prescription is a critical nephrology skill. We administered a formative objective structured clinical examination (OSCE) to nephrology fellows to assess acute KRT medical knowledge, patient care, and systems-based practice competencies.

Design, setting, participants, & measurements

Prospective cohort study of an educational test using the unified model of construct validity. We tested 117 fellows: 25 (four programs) in 2016 and 92 (15 programs) in 2017; 51 first-year and 66 second-year fellows. Using institutional protocols and order sets, fellows wrote orders and answered open-ended questions on a three-scenario OSCE, previously validated by board-certified, practicing clinical nephrologists. Outcomes were overall and scenario pass percentage and score; percent correctly answering predetermined, evidence-based questions; second-year score correlation with in-training examination score; and satisfaction survey.

Results

A total of 76% passed scenario 1 (acute continuous RRT): 92% prescribed a ≥20 ml/kg per hour effluent dose; 63% estimated clearance as effluent volume. Forty-two percent passed scenario 2 (maintenance dialysis initiation); 75% correctly prescribed 3-4 mEq/L K+ dialysate and 12% identified the two absolute, urgent indications for maintenance dialysis initiation (uremic encephalopathy and pericarditis). Six percent passed scenario 3 (acute life-threatening hyperkalemia); 20% checked for rebound hyperkalemia with two separate blood draws. Eighty-three percent correctly withheld intravenous sodium bicarbonate for acute hyperkalemia in a nonacidotic, volume-overloaded patient on maintenance dialysis, and 32% passed overall. Second-year versus first-year fellow overall score was 44.4±4 versus 42.7±5 (one-tailed P=0.02), with 39% versus 24% passing (P=0.08). Second-year in-training examination and OSCE scores were not significantly correlated (r=0.15; P=0.26). Seventy-seven percent of fellows agreed the OSCE was useful in assessing "proficiency in ordering" acute KRT. Limitations include lack of a validated criterion test, and unfamiliarity with open-ended question format.

Conclusions

The OSCE can provide quantitative data for formative Accreditation Council for Graduate Medical Education competency assessments and identify opportunities for dialysis curriculum development.

Podcast

This article contains a podcast at https://www.asn-online.org/media/podcast/CJASN/2019_08_08_CJASNPodcast_19_09_.mp3.

SUBMITTER: Prince LK

PROVIDER: S-EPMC6730513 | biostudies-literature | 2019 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The Acute Dialysis Orders Objective Structured Clinical Examination (OSCE): Fellow Performance on a Formative Assessment of Acute Kidney Replacement Therapy Competence.

Prince Lisa K LK Nee Robert R Yuan Christina M CM

Clinical journal of the American Society of Nephrology : CJASN 20190813 9

<h4>Background and objectives</h4>Acute kidney replacement therapy (KRT) prescription is a critical nephrology skill. We administered a formative objective structured clinical examination (OSCE) to nephrology fellows to assess acute KRT medical knowledge, patient care, and systems-based practice competencies.<h4>Design, setting, participants, & measurements</h4>Prospective cohort study of an educational test using the unified model of construct validity. We tested 117 fellows: 25 (four programs) ...[more]

PMID: 31409597

Similar Datasets

Project description:PurposeEnsuring that examiners in different parallel circuits of objective structured clinical examinations (OSCEs) judge to the same standard is critical to the chain of validity. Recent work suggests examiner-cohort (i.e., the particular group of examiners) could significantly alter outcomes for some candidates. Despite this, examiner-cohort effects are rarely examined since fully nested data (i.e., no crossover between the students judged by different examiner groups) limit comparisons. In this study, the authors aim to replicate and further develop a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA), so it can be used to enhance quality assurance of distributed or national OSCEs.MethodIn 2019, 6 volunteer students were filmed on 12 stations in a summative OSCE. In addition to examining live student performances, examiners from 8 separate examiner-cohorts scored the pool of video performances. Examiners scored videos specific to their station. Video scores linked otherwise fully nested data, enabling comparisons by Many Facet Rasch Modeling. Authors compared and adjusted for examiner-cohort effects. They also compared examiners' scores when videos were embedded (interspersed between live students during the OSCE) or judged later via the Internet.ResultsHaving accounted for differences in students' ability, different examiner-cohort scores for the same ability of student ranged from 18.57 of 27 (68.8%) to 20.49 (75.9%), Cohen's d = 1.3. Score adjustment changed the pass/fail classification for up to 16% of students depending on the modeled cut score. Internet and embedded video scoring showed no difference in mean scores or variability. Examiners' accuracy did not deteriorate over the 3-week Internet scoring period.ConclusionsExaminer-cohorts produced a replicable, significant influence on OSCE scores that was unaccounted for by typical assessment psychometrics. VESCA offers a promising means to enhance validity and fairness in distributed OSCEs or national exams. Internet-based scoring may enhance VESCA's feasibility.

Project description:This study aimed to investigate the impact of student e-learning on the development of clinical competencies. The study participants were 3rd year students (n = 43) at a private mid-sized medical school located in a South Korean suburb on a four-year medical program. Educational intervention was implemented to enhance student clinical performance. Students engaged in learning activities that intended to promote their self-directed learning abilities and clinical performances using e-learning resources. Intervention was conducted for the duration of six months during the 3rd year and its effectiveness was investigated by comparing student performances in OSCEs in a pre- and post- comparison format and also by comparing them with national scores. In addition, student perceptions of the impact of e-learning on their OSCE performances were assessed using a questionnaire, which included 36 items that elicited student perceptions of their experiences of e-learning and readiness for e-learning. Student OSCE scores improved significantly after educational intervention in all domains of clinical competencies assessed and for total scores (p < 0.001). Furthermore, students achieved higher OSCE scores than national average scores in the post-test, whereas they had performed lower than national average scores in the pre-test. Students showed neutral or slightly positive responses to the effectiveness of e-learning, and their perceptions of e-learning were not associated with their e-learning readiness scores. The study shows student OSCE performance improved significantly after educational intervention, which indicate the effectiveness of e-learning to support student learning of clinical performance. Despite significant improvements in student OSCE scores after e-learning, their perceptions of its effectiveness were neutral. Furthermore, student perceptions of e-learning were not associated with their readiness for it. Suggestions are made to help students use e-learning more effectively to enhance their clinical competencies.

Project description:BackgroundObjective structured clinical examinations (OSCEs) are a widely recognized and accepted method to assess clinical competencies but are often resource-intensive.ObjectiveThis study aimed to evaluate the feasibility and effectiveness of a virtual reality (VR)-based station (VRS) compared with a traditional physical station (PHS) in an already established curricular OSCE.MethodsFifth-year medical students participated in an OSCE consisting of 10 stations. One of the stations, emergency medicine, was offered in 2 modalities: VRS and PHS. Students were randomly assigned to 1 of the 2 modalities. We used 2 distinct scenarios to prevent content leakage among participants. Student performance and item characteristics were analyzed, comparing the VRS with PHS as well as with 5 other case-based stations. Student perceptions of the VRS were collected through a quantitative and qualitative postexamination online survey, which included a 5-point Likert scale ranging from 1 (minimum) to 5 (maximum), to evaluate the acceptance and usability of the VR system. Organizational and technical feasibility as well as cost-effectiveness were also evaluated.ResultsFollowing randomization and exclusions of invalid data sets, 57 and 66 participants were assessed for the VRS and PHS, respectively. The feasibility evaluation demonstrated smooth implementation of both VR scenarios (septic and anaphylactic shock) with 93% (53/57) of students using the VR technology without issues. The difficulty levels of the VRS scenarios (septic shock: P=.67; anaphylactic shock: P=.58) were comparable to the average difficulty of all stations (P=.68) and fell within the reference range (0.4-0.8). In contrast, VRS demonstrated above-average values for item discrimination (septic shock: r'=0.40; anaphylactic shock: r'=0.33; overall r'=0.30; with values >0.3 considered good) and discrimination index (septic shock: D=0.25; anaphylactic shock: D=0.26; overall D=0.16, with 0.2-0.3 considered mediocre and <0.2 considered poor). Apart from some hesitancy toward its broader application in future practical assessments (mean 3.07, SD 1.37 for VRS vs mean 3.65, SD 1.18 for PHS; P=.03), there were no other differences in perceptions between VRS and PHS. Thematic analysis highlighted the realistic portrayal of medical emergencies and fair assessment conditions provided by the VRS. Regarding cost-effectiveness, initial development of the VRS can be offset by long-term savings in recurring expenses like standardized patients and consumables.ConclusionsIntegration of the VRS into the current OSCE framework proved feasible both technically and organizationally, even within the strict constraints of short examination phases and schedules. The VRS was accepted and positively received by students across various levels of technological proficiency, including those with no prior VR experience. Notably, the VRS demonstrated comparable or even superior item characteristics, particularly in terms of discrimination power. Although challenges remain, such as technical reliability and some acceptance concerns, VR remains promising in applications of clinical competence assessment.

Project description:BackgroundClinical education is a significant part of medical education, and paying attention to clinical evaluation is important. One of the challenges in the teaching-learning process is evaluating students' performance. Learners, as the main stakeholders of the educational system, may have different experiences of evaluation quality. Awareness of these experiences is effective in improving the quality of clinical evaluation. Therefore, this study was conducted to "explore the experiences of nursing students for evaluation of professional competence by using the objective structured clinical examination (OSCE)".MethodThis study was conducted with a qualitative descriptive research approach and conventional content analysis method in 2022-2024. The participants included 12 undergraduate nursing students at Maragheh University of Medical Sciences, who were selected by purposeful sampling, and their experiences were collected using semi-structured and in-depth interviews until reaching data saturation.ResultsThe data analysis of the interviews led to the extraction of 268 primary codes, 7 subcategories, 2 categories, and 1 theme: "Credibility and stability". The category "Exam's accuracy in measuring competence " included 3 sub-categories: "Challenges in objective adaptation ", " Communication and organizational challenges for exam preparation " and " Inadequate simulation of stations and exam environment ", and the category " Exam power for repeatability" included 4 sub-categories: " Characteristics of the students ", " Lack of evaluators' skills and mastery ", " Inefficiency of the evaluation tool " and " Disturbance in executive affairs".ConclusionsOSCE can be used in self-evaluation, creating motivation and strengthening different dimensions of students' learning, as well as discovering weaknesses and strengths for planning by managers and faculties. According to the results of this study, many factors such as management before and during the exam, characteristics of the evaluators, prevailing educational conditions in the faculty, and the method of clinical training are effective in achieving the "reliability and sustainability" required in the OSCE.

Dataset Information

The Acute Dialysis Orders Objective Structured Clinical Examination (OSCE): Fellow Performance on a Formative Assessment of Acute Kidney Replacement Therapy Competence.

Background and objectives

Design, setting, participants, & measurements

Results

Conclusions

Podcast

Publications

The Acute Dialysis Orders Objective Structured Clinical Examination (OSCE): Fellow Performance on a Formative Assessment of Acute Kidney Replacement Therapy Competence.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets