An Objective Structured Clinical Examination to Improve Formative Assessment for Senior Pediatrics Residents.
Ontology highlight
ABSTRACT: Background
Residency programs are developing new methods to assess resident competence and to improve the quality of formative assessment and feedback to trainees. Simulation is a valuable tool for giving formative feedback to residents.Objective
To develop an objective structured clinical examination (OSCE) to improve formative assessment of senior pediatrics residents.Methods
We developed a multistation examination using various simulation formats to assess the skills of senior pediatrics residents in communication and acute resuscitation. We measured several logistical factors (staffing and program costs) to determine the feasibility of such a program.Results
Thirty-one residents participated in the assessment program over a 3-month period. Residents received formative feedback comparing their performance to both a standard task checklist and to peers' performance. The program required 16 faculty members per session, and had a cost of $624 per resident.Conclusions
A concentrated assessment program using simulation can be a valuable tool to assess residents' skills in communication and acute resuscitation and provide directed formative feedback. However, such a program requires considerable financial and staffing resources.
SUBMITTER: Mangold KA
PROVIDER: S-EPMC4597964 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundFew quantitative nephrology-specific simulations assess fellow competency. We describe the development and initial validation of a formative objective structured clinical examination (OSCE) assessing fellow competence in ordering acute dialysis.MethodsThe three test scenarios were acute continuous renal replacement therapy, chronic dialysis initiation in moderate uremia and acute dialysis in end-stage renal disease-associated hyperkalemia. The test committee included five academic nephrologists and four clinically practicing nephrologists outside of academia. There were 49 test items (58 points). A passing score was 46/58 points. No item had median relevance less than 'important'. The content validity index was 0.91. Ninety-five percent of positive-point items were easy-medium difficulty. Preliminary validation was by 10 board-certified volunteers, not test committee members, a median of 3.5 years from graduation. The mean score was 49 [95% confidence interval (CI) 46-51], κ = 0.68 (95% CI 0.59-0.77), Cronbach's α = 0.84.ResultsWe subsequently administered the test to 25 fellows. The mean score was 44 (95% CI 43-45); 36% passed the test. Fellows scored significantly less than validators (P < 0.001). Of evidence-based questions, 72% were answered correctly by validators and 54% by fellows (P = 0.018). Fellows and validators scored least well on the acute hyperkalemia question. In self-assessing proficiency, 71% of fellows surveyed agreed or strongly agreed that the OSCE was useful.ConclusionsThe OSCE may be used to formatively assess fellow proficiency in three common areas of acute dialysis practice. Further validation studies are in progress.
| S-EPMC5887504 | biostudies-literature
Project description:Background and objectivesAcute kidney replacement therapy (KRT) prescription is a critical nephrology skill. We administered a formative objective structured clinical examination (OSCE) to nephrology fellows to assess acute KRT medical knowledge, patient care, and systems-based practice competencies.Design, setting, participants, & measurementsProspective cohort study of an educational test using the unified model of construct validity. We tested 117 fellows: 25 (four programs) in 2016 and 92 (15 programs) in 2017; 51 first-year and 66 second-year fellows. Using institutional protocols and order sets, fellows wrote orders and answered open-ended questions on a three-scenario OSCE, previously validated by board-certified, practicing clinical nephrologists. Outcomes were overall and scenario pass percentage and score; percent correctly answering predetermined, evidence-based questions; second-year score correlation with in-training examination score; and satisfaction survey.ResultsA total of 76% passed scenario 1 (acute continuous RRT): 92% prescribed a ≥20 ml/kg per hour effluent dose; 63% estimated clearance as effluent volume. Forty-two percent passed scenario 2 (maintenance dialysis initiation); 75% correctly prescribed 3-4 mEq/L K+ dialysate and 12% identified the two absolute, urgent indications for maintenance dialysis initiation (uremic encephalopathy and pericarditis). Six percent passed scenario 3 (acute life-threatening hyperkalemia); 20% checked for rebound hyperkalemia with two separate blood draws. Eighty-three percent correctly withheld intravenous sodium bicarbonate for acute hyperkalemia in a nonacidotic, volume-overloaded patient on maintenance dialysis, and 32% passed overall. Second-year versus first-year fellow overall score was 44.4±4 versus 42.7±5 (one-tailed P=0.02), with 39% versus 24% passing (P=0.08). Second-year in-training examination and OSCE scores were not significantly correlated (r=0.15; P=0.26). Seventy-seven percent of fellows agreed the OSCE was useful in assessing "proficiency in ordering" acute KRT. Limitations include lack of a validated criterion test, and unfamiliarity with open-ended question format.ConclusionsThe OSCE can provide quantitative data for formative Accreditation Council for Graduate Medical Education competency assessments and identify opportunities for dialysis curriculum development.PodcastThis article contains a podcast at https://www.asn-online.org/media/podcast/CJASN/2019_08_08_CJASNPodcast_19_09_.mp3.
| S-EPMC6730513 | biostudies-literature
Project description:Internal medicine residents care for a sizable number of patients with chronic pain. Programs need educational strategies to promote safe opioid prescribing.To describe a safe opioid prescribing education program utilizing an objective structured clinical examination (OSCE) and report the resulting impact on residents' knowledge, confidence, and self-reported practices.Using a quasi-experimental design, 39 internal medicine residents from an urban academic medical center were assigned to 1 of 4 groups: 1-hour lecture only, lecture followed by immediate OSCE, lecture followed by 4-month delayed OSCE, and control. Safe opioid prescribing knowledge, confidence, and self-reported practices were assessed at baseline and at 8 months.At 8 months, knowledge, confidence, and self-reported practices improved in the control and in all 3 intervention groups. The immediate OSCE group had the greatest improvements in combined confidence scores within group (0.74, P?=?.01) compared to controls (0.52, P?=?.05), using a 5-point scale. This group also had the greatest improvement in self-reported practice changes (1.04, P?=?.04), while other groups showed nonsignificant improvements-delayed OSCE (0.43, P?=?.44), lecture only (0.66, P?=?.24), and control (0.43, P?=?.19).Safe opioid prescribing education that includes a lecture immediately followed by an OSCE had an impact on residents' confidence and self-reported practices greater than those for delayed OSCE or lecture only groups. There was no difference in knowledge improvement among the groups. Lecture followed by an OSCE was highly regarded by residents, but required additional resources.
| S-EPMC4936858 | biostudies-other
Project description:ObjectivesWe evaluated the reliability and predictive ability of an objective structured clinical examination (OSCE) in the assessment of medical students at the completion of a neurology clerkship.MethodsWe analyzed data from 195 third-year medical students who took the OSCE. For each student, the OSCE consisted of 2 standardized patient encounters. The scores obtained from each encounter were compared. Faculty clinical evaluations of each student for 2 clinical inpatient rotations were also compared. Hierarchical regression analysis was applied to test the ability of the averaged OSCE scores to predict standardized written examination scores and composite clinical scores.ResultsStudents' OSCE scores from the 2 standardized patient encounters were significantly correlated with each other (r = 0.347, p < 0.001), and the scores for all students were normally distributed. In contrast, students' faculty clinical evaluation scores from 2 different clinical inpatient rotations were uncorrelated, and scores were skewed toward the highest ratings. After accounting for clerkship order, better OSCE scores were predictive of better National Board of Medical Examiners standardized examination scores (R(2)? = 0.131, p < 0.001) and of better faculty clinical scores (R(2)? = 0.078, p < 0.001).ConclusionsStudent assessment by an OSCE provides a reliable and predictive objective assessment of clinical performance in a neurology clerkship.
| S-EPMC3414663 | biostudies-literature
Project description:ObjectivesThis study examines factors affecting reliability, or consistency of assessment scores, from an objective structured clinical examination (OSCE) in neurology through generalizability theory (G theory).MethodsData include assessments from a multistation OSCE taken by 194 medical students at the completion of a neurology clerkship. Facets evaluated in this study include cases, domains, and items. Domains refer to areas of skill (or constructs) that the OSCE measures. G theory is used to estimate variance components associated with each facet, derive reliability, and project the number of cases required to obtain a reliable (consistent, precise) score.ResultsReliability using G theory is moderate (Φ coefficient = 0.61, G coefficient = 0.64). Performance is similar across cases but differs by the particular domain, such that the majority of variance is attributed to the domain. Projections in reliability estimates reveal that students need to participate in 3 OSCE cases in order to increase reliability beyond the 0.70 threshold.ConclusionsThis novel use of G theory in evaluating an OSCE in neurology provides meaningful measurement characteristics of the assessment. Differing from prior work in other medical specialties, the cases students were randomly assigned did not influence their OSCE score; rather, scores varied in expected fashion by domain assessed.
| S-EPMC4642144 | biostudies-literature
Project description:BackgroundObjective structured clinical examination (OSCE) has been used in evaluating clinical competence in health professions education around the world. Despite its implementation in Iraq for around a decade, limited studies investigated the challenges and opportunities to improve the standard and quality of this examination from student's perspective.MethodsThis qualitative study was based on an online open-ended questionnaire survey that was carried out in the College of Medicine, Hawler Medical University, Iraq at the beginning of the 2018-2019 academic year. A convenience sample of 180 students in the clinical phase (4th, 5th, and 6th) year of study were invited to participate.ResultsA total of 141 students responded to the online questionnaire. The participants were generally happy with the OSCE, and they recognized many positive aspects, including the role of the OSCE in increasing confidence, engagement and motivating learning, the role of the OSCE in achieving a higher level of learning, the content validity of the OSCE, and the quality of the OSCE. The main weak points of the OSCE identified by the students included unfairness, gender discrimination, duration of the OSCE, and the behavior of the examiners. Suggestions to improve the OSCE examination included improving the examiners' behavior, with the focus on the training of the examiners, and avoiding discrimination among students.ConclusionsMost of the students were generally satisfied with the current OSCE examination. The main concern of the students was related to the organization of the OSCE. Valuable suggestions were raised to improve the OSCE quality including examiners' and simulated patients' training.
| S-EPMC9439190 | biostudies-literature
Project description:PURPOSE:This case study explored the use of Google Glass in a clinical examination scenario to capture the first-person perspective of a standardized patient as a way to provide formative feedback on students' communication and empathy skills 'through the patient's eyes.' METHODS:During a 3-year period between 2014 and 2017, third-year students enrolled in a family medicine clerkship participated in a Google Glass station during a summative clinical examination. At this station, standardized patients wore Google Glass to record an encounter focused on communication and empathy skills 'through the patient's eyes.' Students completed an online survey using a 4-point Likert scale about their perspectives on Google Glass as a feedback tool (N= 255). RESULTS:We found that the students' experiences with Google Glass 'through the patient's eyes' were largely positive and that students felt the feedback provided by the Google Glass recording to be helpful. Although a third of the students felt that Google Glass was a distraction, the majority believed that the first-person perspective recordings provided an opportunity for feedback that did not exist before. CONCLUSION:Continuing exploration of first-person perspective recordings using Google Glass to improve education on communication and empathy skills is warranted.
| S-EPMC5900380 | biostudies-literature
Project description:PurposeIt aimed to compare the use of the tele objective structured clinical examination (teleOSCE) with in-person assessment in high-stakes clinical examination so as to determine the impact of the teleOSCE on the assessment undertaken. Discussion follows regarding what skills and domains can effectively be assessed in a teleOSCE.MethodsThis study is a retrospective observational analysis. It compares the results achieved by final year medical students in their clinical examination, assessed using the teleOSCE in 2020 (n=285), with those who were examined using the traditional in-person format in 2019 (n=280). The study was undertaken at the University of New South Wales, Australia.ResultsIn the domain of physical examination, students in 2020 scored 0.277 points higher than those in 2019 (mean difference -0.277, P<0.001, effect size 0.332). Across all other domains, there was no significant difference in mean scores between 2019 and 2020.ConclusionThe teleOSCE does not negatively impact assessment in clinical examination in all domains except physical examination. If the teleOSCE is the future of clinical skills examination, assessment of physical examination will require concomitant workplace-based assessment.
| S-EPMC8616724 | biostudies-literature
Project description:Introduction: The aim of the Objective Structured Clinical Examination (OSCE) is a standardized and fair assessment of clinical skills. Observing second clinical year medical students during a summative OSCE assessing a General Practice clerkship, we noticed that information exchange with peers led to a progressively faster and overly focused management of simulations. Therefore, we established a Multiple Scenario-OSCE (MS-OSCE) where all students had to manage the same chief complaint at a station but it's underlying scenarios being randomly changed during students' rotation through their parcours. We wanted to ensure they fully explore differential diagnosis instead of managing their task influenced by shared information. We wanted to assess if a MS-OSCE violates the assumption of objectivity and fairness given that students are not tested with the same scenarios. Methods: We developed and piloted five OSCE stations (chest pain, abdominal pain, back pain, fatigue and acute cough) with two or three different underlying scenarios each. At each station these scenarios randomly changed from student to student. Performance was assessed with a checklist and global rating. The effect of scenarios and raters on students' grades was assessed calculating the intraclass correlation coefficient with a fixed effect two level linear model. Results: A total of 169 students and 23 raters participated in the MS-OSCE. The internal consistency over all stations was 0.65 by Cronbach's alpha. The difference of the mean grades between the scenarios of a given chief complaint ranged from 0.03 to 0.4 on a 1 to 5 grading scale. The effect of scenarios on the variance of the final grades at each station ranged from 4% to 9% and of raters from 20% to 50% when adjusted for students' skills. Conclusions: The effect of different scenarios on the grades was relevant but small compared to the effect of raters on grades. Improving rater training is more important to ensure objectivity and fairness of MS-OSCE than providing the same scenario to all students.
| S-EPMC6545613 | biostudies-literature
Project description:OBJECTIVE: To investigate if the psychometric qualities of an OSCE consisting of more complex simulated patient encounters remain valid and reliable in the assessment of postgraduate trainees in general practice. METHODS: In this intervention study without control group, the traditional OSCE was formally replaced by the new, complex version. The study population was composed by all postgraduate trainees (second and third phase) in general practice during the ongoing academic year. Data were handled and collected as part of the formal assessment program. Univariate analyses, the variance of scores and multivariate analyses were performed to assess the test qualities. RESULTS: A total of 340 students participated. Average final scores were slightly higher for third-phase students (t-test, p =0.05). Overall test scores were equally distributed on station level, circuit level and phase level. A multiple regression analysis revealed that test scores were dependent on the stations and circuits, but not on the master phase. CONCLUSIONS: In a changing learning environment, assessment and evaluation strategies require reorientation. The reliability and validity of the OSCE remain subject to discussion. In particular, when it comes to content and design, the traditional OSCE might underestimate the performance level of postgraduate trainees in general practice. A reshaping of this OSCE to a more sophisticated design with more complex patient encounters appears to restore the validity of the test results.
| S-EPMC4224044 | biostudies-other