Dataset Information

Comparison of Simulation-based Resuscitation Performance Assessments With In-training Evaluation Reports in Emergency Medicine Residents: A Canadian Multicenter Study.

ABSTRACT: Simulation stands to serve an important role in modern competency-based programs of assessment in postgraduate medical education. Our objective was to compare the performance of individual emergency medicine (EM) residents in a simulation-based resuscitation objective structured clinical examination (OSCE) using the Queen's Simulation Assessment Tool (QSAT), with portfolio assessment of clinical encounters using a modified in-training evaluation report (ITER) to understand in greater detail the inferences that may be drawn from a simulation-based OSCE assessment.A prospective observational study was employed to explore the use of a multicenter simulation-based OSCE for evaluation of resuscitation competence. EM residents from five Canadian academic sites participated in the OSCE. Video-recorded performances were scored by blinded raters using the scenario-specific QSATs with domain-specific anchored scores (primary assessment, diagnostic actions, therapeutic actions, communication) and a global assessment score (GAS). Residents' portfolios were evaluated using a modified ITER subdivided by CanMEDS roles (medical expert, communicator, collaborator, leader, health advocate, scholar, and professional) and a GAS. Correlational and regression analyses were performed comparing components of each of the assessment methods.Portfolio review and ITER scoring was performed for 79 residents participating in the simulation-based OSCE. There was a significant positive correlation between total OSCE and ITER scores (r = 0.341). The strongest correlations were found between ITER medical expert score and each of the OSCE GAS (r = 0.420), communication (r = 0.443), and therapeutic action (r = 0.484) domains. ITER medical expert was a significant predictor of OSCE total (p = 0.002). OSCE therapeutic action was a significant predictor of ITER total (p = 0.02).Simulation-based resuscitation OSCEs and portfolio assessment captured by ITERs appear to measure differing aspects of competence, with weak to moderate correlation between those measures of conceptually similar constructs. In a program of competency-based assessment of EM residents, a simulation-based OSCE using the QSAT shows promise as a tool for assessing medical expert and communicator roles.

SUBMITTER: Hall AK

PROVIDER: S-EPMC6001706 | biostudies-other | 2017 Oct

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Comparison of Simulation-based Resuscitation Performance Assessments With In-training Evaluation Reports in Emergency Medicine Residents: A Canadian Multicenter Study.

Hall Andrew Koch AK Damon Dagnone J J Moore Sean S Woolfrey Karen G H KGH Ross John A JA McNeil Gordon G Hagel Carly C Davison Colleen C Sebok-Syer Stefanie S SS

AEM education and training 20170919 4

<h4>Objective</h4>Simulation stands to serve an important role in modern competency-based programs of assessment in postgraduate medical education. Our objective was to compare the performance of individual emergency medicine (EM) residents in a simulation-based resuscitation objective structured clinical examination (OSCE) using the Queen's Simulation Assessment Tool (QSAT), with portfolio assessment of clinical encounters using a modified in-training evaluation report (ITER) to understand in g ...[more]

PMID: 30051047

Similar Datasets

Project description:Simulation-based teaching (SIM) is a common method for medical education. SIM exposes residents to uncommon scenarios that require critical, timely actions. SIM may be a valuable training method for critically ill poisoned patients whose diagnosis and treatment depend on key clinical findings. Our objective was to compare medical simulation (SIM) to traditional lecture-based instruction (LEC) for training emergency medicine (EM) residents in the acute management of critically ill poisoned patients. EM residents completed two pre-intervention questionnaires: (1) a 24-item multiple-choice test of four toxicological emergencies and (2) a questionnaire using a five-point Likert scale to rate the residents' comfort level in diagnosing and treating patients with specific toxicological emergencies. After completing the pre-intervention questionnaires, residents were randomized to SIM or LEC instruction. Two toxicologists and three EM physicians presented four toxicology topics to both groups in four 20-min sessions. One group was in the simulation center, and the other in a lecture hall. Each group then repeated the multiple-choice test and questionnaire immediately after instruction and again at 3 months after training. Answers were not discussed. The primary outcome was comparison of immediate mean post-intervention test scores and final scores 3 months later between SIM and LEC groups. Test score outcomes between groups were compared at each time point (pre-test, post-instruction, 3-month follow-up) using Wilcoxon rank sum test. Data were summarized by descriptive statistics. Continuous variables were characterized by means (SD) and tested using t tests or Wilcoxon rank sum. Categorical variables were summarized by frequencies (%) and compared between training groups with chi-square or Fisher's exact test. Thirty-two EM residents completed pre- and post-intervention tests and comfort questionnaires on the study day. Both groups had higher post-intervention mean test scores (p?<?0.001), but the LEC group showed a greater improvement compared to the SIM group (5.6 [2.3] points vs. 3.6 [2.4], p?=?0.02). At the 3-month follow-up, 24 (75 %) tests and questionnaires were completed. There was no improvement in 3-month mean test scores in either group compared to immediate post-test scores. The SIM group had higher final mean test scores than the LEC group (16.6 [3.1] vs. 13.3 [2.2], p?=?0.009). SIM and LEC groups reported similar diagnosis and treatment comfort level scores at baseline and improved equally after instruction. At 3 months, there was no difference between groups in comfort level scores for diagnosis or treatment. Lecture-based teaching was more effective than simulation-based instruction immediately after intervention. At 3 months, the SIM group showed greater retention than the LEC group. Resident comfort levels for diagnosis and treatment were similar regardless of the type of education.

Project description:BackgroundThe majority of children in the United States seek emergency care at community-based general emergency departments (GEDs); however, the quality of GED pediatric emergency care varies widely. This may be explained by a number of factors, including residency training environments and postgraduate knowledge decay. Emergency medicine (EM) residents train in academic pediatric EDs, but didactic and clinical experience vary widely between programs, and little is known about the pediatric skills of these EM residents. This study aimed to assess the performance of senior EM residents in treating simulated pediatric patients at the end of their training.MethodsThis was a prospective, cross-sectional, simulation-based cohort study assessing the simulated performance of senior EM resident physicians from two Massachusetts programs leading medical teams caring for three critically ill patients. Sessions were video recorded and scored separately by three reviewers using a previously published simulation assessment tool. Self-efficacy surveys were completed prior to each session. The primary outcome was a median total performance score (TPS), calculated by the mean of individualized domain scores (IDS) for each case. Each IDS was calculated as a percentage of items performed on a checklist-based instrument.ResultsA total of 18 EM resident physicians participated (PGY-3 = 8, PGY-4 = 10). Median TPS for the cohort was 61% (IQR = 56%-70%). Median IDSs by case were as follows: sepsis 67% (IQR = 50%-67%), seizure 67% (IQR = 50%-83%), and cardiac arrest 67% (IQR = 43%-70%). The overall cohort self-efficacy for pediatric EM (PEM) was 64% (IQR = 60%-70%).ConclusionsThis study has begun the process of benchmarking clinical performance of graduating EM resident physicians. Overall, the EM resident cohort in this study performed similar to prior GED teams. Self-efficacy related to PEM correlated well with performance, with the exception of knowledge relative to intravenous fluid and vasopressor administration in pediatric septic shock. A significant area of discrepancy and missed checklist items were those related to cardiopulmonary resuscitation and basic life support maneuvers.

Dataset Information

Comparison of Simulation-based Resuscitation Performance Assessments With In-training Evaluation Reports in Emergency Medicine Residents: A Canadian Multicenter Study.

Publications

Comparison of Simulation-based Resuscitation Performance Assessments With In-training Evaluation Reports in Emergency Medicine Residents: A Canadian Multicenter Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure