Dataset Information

The interrater reliability of an objective structured practical examination in measuring the clinical reasoning ability of chiropractic students.

ABSTRACT: OBJECTIVE:The objective structured practical examination (OSPE) is a case-based assessment that can be used to assess the clinical reasoning ability of students. The reliability of using an OSPE for this purpose has not been reported in the literature. The objective of this study was to determine the interrater reliability of the OSPE in measuring the clinical reasoning ability of chiropractic students. METHODS:Two examiners tested each student simultaneously when enough were available as a check for interrater reliability. The scores for students over 4 exam administrations were compiled, and we calculated an intraclass correlation coefficient (ICC) using 1-way random single measures. RESULTS:Paired scores were available for 133 students. The ICC was .685, showing a fair-to-good level of agreement for faculty in assessing the clinical reasoning ability of chiropractic students using an OSPE. CONCLUSION:The OSPE can be a valuable tool for testing clinical reasoning abilities because it can simulate the decision-making process that needs to be implemented in clinical practice. Faculty members at our chiropractic college were able to achieve an acceptable level of reliability in measuring the clinical reasoning abilities of students using an OSPE. Other health professional programs may consider using this tool for assessing the clinical reasoning skills of their students.

SUBMITTER: Rose KA

PROVIDER: S-EPMC5067129 | biostudies-literature | 2016 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The interrater reliability of an objective structured practical examination in measuring the clinical reasoning ability of chiropractic students.

Rose Kevin A KA Babajanian Jesika J

The Journal of chiropractic education 20160426 2

<h4>Objective</h4>The objective structured practical examination (OSPE) is a case-based assessment that can be used to assess the clinical reasoning ability of students. The reliability of using an OSPE for this purpose has not been reported in the literature. The objective of this study was to determine the interrater reliability of the OSPE in measuring the clinical reasoning ability of chiropractic students.<h4>Methods</h4>Two examiners tested each student simultaneously when enough were avai ...[more]

PMID: 27115474

Similar Datasets

Project description:PurposeEnsuring that examiners in different parallel circuits of objective structured clinical examinations (OSCEs) judge to the same standard is critical to the chain of validity. Recent work suggests examiner-cohort (i.e., the particular group of examiners) could significantly alter outcomes for some candidates. Despite this, examiner-cohort effects are rarely examined since fully nested data (i.e., no crossover between the students judged by different examiner groups) limit comparisons. In this study, the authors aim to replicate and further develop a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA), so it can be used to enhance quality assurance of distributed or national OSCEs.MethodIn 2019, 6 volunteer students were filmed on 12 stations in a summative OSCE. In addition to examining live student performances, examiners from 8 separate examiner-cohorts scored the pool of video performances. Examiners scored videos specific to their station. Video scores linked otherwise fully nested data, enabling comparisons by Many Facet Rasch Modeling. Authors compared and adjusted for examiner-cohort effects. They also compared examiners' scores when videos were embedded (interspersed between live students during the OSCE) or judged later via the Internet.ResultsHaving accounted for differences in students' ability, different examiner-cohort scores for the same ability of student ranged from 18.57 of 27 (68.8%) to 20.49 (75.9%), Cohen's d = 1.3. Score adjustment changed the pass/fail classification for up to 16% of students depending on the modeled cut score. Internet and embedded video scoring showed no difference in mean scores or variability. Examiners' accuracy did not deteriorate over the 3-week Internet scoring period.ConclusionsExaminer-cohorts produced a replicable, significant influence on OSCE scores that was unaccounted for by typical assessment psychometrics. VESCA offers a promising means to enhance validity and fairness in distributed OSCEs or national exams. Internet-based scoring may enhance VESCA's feasibility.

Project description:BackgroundThe physical examination is one of the cornerstones of the diagnostic process in patients with acute shoulder injuries. The discriminative properties of a given examination test depend both on its validity and reliability. The aim of the present study was to assess the interrater reliability of 13 physical examination manoeuvres for acute rotator cuff tears in patients with acute soft tissue shoulder injuries.MethodsIn a large walk-in orthopaedic emergency department, 120 consecutive patients ≥40 years of age were included in a diagnostic study. Patients who had follow-up within three weeks of an acute shoulder injury without fracture on radiographs were eligible. Four emergency department physicians participated as examiners. In a subset of 48 patients, the physical examination tests were performed by two physicians, randomly chosen by their work rotation. The physicians were blinded to the findings of each other and the results of the ultrasound screening. The interrater reliability was assessed by Cohen's kappa, intraclass correlation coefficient (ICC), standard error of measurement (SEM) and Bland-Altman plots depending on whether the examination test result was registered as a binary, ordered categorical or continuous numerical variable.ResultsThe median age was 55.5 years, 46% were female. Twenty-seven percent had a rotator cuff full-thickness tear on ultrasound screening; all but one involved the supraspinatus tendon. Cohen's kappa for binary tests ranged from excellent to fair. Excellent agreement (kappa > 0.8) was found for the inability to abduct above 90° and abduction strength. External rotation strength expressed substantial agreement (kappa 0.7). The lowest scores were registered for Hawkins` test and the external rotation lag sign (kappa 0.25 and 0.40, respectively). The ICCs for active range of abduction and external rotation were 0.93 (0.88-0.96) and 0.84 (0.72-0.91), whereas the SEM was 15 and 9, respectively.ConclusionsThe results indicate that examination manoeuvres assessing abduction and external rotation range of motion and strength are more reliable than manoeuvres assessing pain in patients in the acute phase of traumatic shoulder injury. The poor agreement observed is likely to limit the validity in the present setting of two commonly used tests.Trial registrationThe Norwegian Regional Ethics Committee South East ( 2015/195 ).

Project description:Introduction: The aim of the Objective Structured Clinical Examination (OSCE) is a standardized and fair assessment of clinical skills. Observing second clinical year medical students during a summative OSCE assessing a General Practice clerkship, we noticed that information exchange with peers led to a progressively faster and overly focused management of simulations. Therefore, we established a Multiple Scenario-OSCE (MS-OSCE) where all students had to manage the same chief complaint at a station but it's underlying scenarios being randomly changed during students' rotation through their parcours. We wanted to ensure they fully explore differential diagnosis instead of managing their task influenced by shared information. We wanted to assess if a MS-OSCE violates the assumption of objectivity and fairness given that students are not tested with the same scenarios. Methods: We developed and piloted five OSCE stations (chest pain, abdominal pain, back pain, fatigue and acute cough) with two or three different underlying scenarios each. At each station these scenarios randomly changed from student to student. Performance was assessed with a checklist and global rating. The effect of scenarios and raters on students' grades was assessed calculating the intraclass correlation coefficient with a fixed effect two level linear model. Results: A total of 169 students and 23 raters participated in the MS-OSCE. The internal consistency over all stations was 0.65 by Cronbach's alpha. The difference of the mean grades between the scenarios of a given chief complaint ranged from 0.03 to 0.4 on a 1 to 5 grading scale. The effect of scenarios on the variance of the final grades at each station ranged from 4% to 9% and of raters from 20% to 50% when adjusted for students' skills. Conclusions: The effect of different scenarios on the grades was relevant but small compared to the effect of raters on grades. Improving rater training is more important to ensure objectivity and fairness of MS-OSCE than providing the same scenario to all students.

Dataset Information

The interrater reliability of an objective structured practical examination in measuring the clinical reasoning ability of chiropractic students.

Publications

The interrater reliability of an objective structured practical examination in measuring the clinical reasoning ability of chiropractic students.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets