Dataset Information

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring.

ABSTRACT:

Background

Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED.

Methods

The resuscitation assessment tool (RAT) was derived from the previously implemented and studied Queen's simulation assessment tool (QSAT) via a modified expert review process. The RAT uses an anchored global assessment scale to generate an entrustment score and narrative comments. Emergency medicine (EM) residents were assessed using the RAT on cases in simulation-based examinations and in the ED during resuscitation cases from July 2016 to June 2017. Resident mean entrustment scores were compared using Pearson's correlation coefficient to determine the relationship between entrustment in simulation cases and in the ED. Inductive thematic analysis of written commentary was conducted to compare workplace-based with simulation-based feedback.

Results

There was a moderate, positive correlation found between mean entrustment scores in the simulated and workplace-based settings, which was statistically significant (r?=?0.630, n?=?17, p?ConclusionsIn this single-center study with a limited sample size, assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace. This study suggests that resuscitation performance in simulation settings may be an indicator of competence in the clinical setting. However, multiple factors contribute to this complicated and imperfect relationship. It is imperative to consider narrative comments in supporting the rationale for numerical entrustment scores in both settings and to include both simulation and workplace-based assessment in high-stakes decisions of progression.

SUBMITTER: Weersink K

PROVIDER: S-EPMC6492388 | biostudies-literature | 2019

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring.

Weersink Kristen K Hall Andrew K AK Rich Jessica J Szulewski Adam A Dagnone J Damon JD

Advances in simulation (London, England) 20190501

<h4>Background</h4>Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED. ...[more]

PMID: 31061721

Similar Datasets

Project description:BackgroundSimulation provides consistent opportunities for residents to practice high-stakes, low-frequency events such as pediatric resuscitations. To increase standardization across North American residency programs, the Emergency Medicine Resident Simulation Curriculum for Pediatrics (EM ReSCu Peds) was developed. However, access to high-quality simulation/pediatric expertise is not uniform. As the concurrent COVID-19 pandemic necessitated new virtual simulation methods, we adapted the Virtual Resus Room (VRR) to teach EM ReSCu Peds. VRR is an award-winning, low-resource, open-access distance telesimulation platform we hypothesize will be effective and scalable for teaching this curriculum.MethodsEM residents completed six VRR EM ReSCu Peds simulation cases and received immediate facilitator-led teledebriefing. Learners completed retrospective pre-post surveys after each case. Learners and facilitators completed end-of-day surveys. Primary outcomes were learning effectiveness measured by a composite of the Simulation Effectiveness in Teaching Modified (SET-M) tool and self-reported changes in learner comfort with case objectives. Secondary outcome was VRR scalability to teach EM ReSCu Peds using a composite outcome of net promoter scores (NPS), resource utilization, open-text feedback, and technical issues.ResultsLearners reported significantly increased comfort with 95% (54/57) of EM ReSCu Peds-defined case objectives (91% cognitive, 9% psychomotor), with moderate (Cohen's d 0.71, 95% CI 0.67-0.76) overall effect size. SET-M responses indicated simulation effectiveness, particularly with debriefing. Ninety EM residents from three North American residency programs were taught by 59 pediatric faculty from six programs over 4 days-more than possible if simulations were conducted in person. Learners (39) and faculty (68) NPS were above software industry benchmarks (13). Minor, quickly resolved, technical issues were reported by 18% and 29% of learners and facilitators, respectively.ConclusionsLearners and facilitators report that the VRR is an effective and scalable platform to teach EM ReSCu Peds. This low-cost, accessible distance simulation intervention could increase equitable, global access to high-quality pediatric emergency education.

Project description:IntroductionThe Accreditation Council for Graduate Medical Education (ACGME) specifically notes multisource feedback (MSF) as a recommended means of resident assessment in the emergency medicine (EM) Milestones. High-fidelity simulation is an environment wherein residents can receive MSF from various types of healthcare professionals. Previously, the Queen's Simulation Assessment Tool (QSAT) has been validated for faculty to assess residents in five categories: assessment; diagnostic actions; therapeutic actions; interpersonal communication, and overall assessment. We sought to determine whether the QSAT could be used to provide MSF using a standardized simulation case.MethodsProspectively after institutional review board approval, residents from a dual ACGME/osteopathic-approved postgraduate years (PGY) 1-4 EM residency were consented for participation. We developed a standardized resuscitation after overdose case with specific 1-5 Likert anchors used by the QSAT. A PGY 2-4 resident participated in the role of team leader, who completed a QSAT as self-assessment. The team consisted of a PGY-1 peer, an emergency medical services (EMS) provider, and a nurse. Two core faculty were present to administer the simulation case and assess. Demographics were gathered from all participants completing QSATs. We analyzed QSATs by each category and on cumulative score. Hypothesis testing was performed using intraclass correlation coefficients (ICC), with 95% confidence intervals. Interpretation of ICC results was based on previously published definitions.ResultsWe enrolled 34 team leader residents along with 34 nurses. A single PGY-1, a single EMS provider and two faculty were also enrolled. Faculty provided higher cumulative QSAT scores than the other sources of MSF. QSAT scores did not increase with team leader PGY level. ICC for inter-rater reliability for all sources of MSF was 0.754 (0.572-0.867). Removing the self-evaluation scores increased inter-rater reliability to 0.838 (0.733-0.910). There was lesser agreement between faculty and nurse evaluations than from the EMS or peer evaluation.ConclusionIn this single-site cohort using an internally developed simulation case, the QSAT provided MSF with excellent reliability. Self-assessment decreases the reliability of the MSF, and our data suggest self-assessment should not be a component of MSF. Use of the QSAT for MSF may be considered as a source of data for clinical competency committees.

Project description:IntroductionMedical professionalism is a core competency for emergency medicine (EM) trainees; but defining professionalism remains challenging, leading to difficulties creating objectives and performing assessment. Because professionalism is dynamic, culture-specific, and often taught by modeling, an exploration of trainees' perceptions can highlight their educational baseline and elucidate the importance they place on general conventional professionalism domains. To this end, our objective was to assess the relative value EM residents place on traditional components of professionalism.MethodsWe performed a cross-sectional, multi-institutional survey of incoming and graduating EM residents at four programs. The survey was developed using the American Board of Internal Medicine's "Project Professionalism" and the Accreditation Council of Graduate Medical Education definition of professionalism competency. We identified 27 attributes within seven domains: clinical excellence, humanism, accountability, altruism, duty and service, honor and integrity, and respect for others. Residents were asked to rate each attribute on a 10-point scale. We analyzed data to assess variance across attributes as well as differences between residents at different training levels or different institutions.ResultsOf the 114 residents eligible, 100 (88%) completed the survey. The relative value assigned to different professional attributes varied considerably, with those in the altruism domain valued significantly lower and those in the "respect for others" and "honor and integrity" valued significantly higher (p<0.001). Significant differences were found between interns and seniors for five attributes primarily in the "duty and service" domain (p<0.05). Among different residencies, significant differences were found with attributes within the "altruism" and "duty and service" domains (p<0.05).ConclusionResidents perceive differences in the relative importance of traditionally defined professional attributes and this may be useful to educators. Explanations for these differences are hypothesized, as are the potential implications for professionalism education. Because teaching professional behavior is taught most effectively via behavior modeling, faculty awareness of resident values and faculty development to address potential gaps may improve professionalism education.

Project description:BackgroundThe majority of children seeking care in emergency departments are seen by general emergency medicine (EM) residency program graduates. Throughout training, EM residents manage fewer critically ill pediatric patients compared to adults, and the exposure to children with illness and injury requiring emergent assessment and management is often limited and sporadic across training sites. This report describes the creation of a robust set of simulation cases for EM trainees incorporating topics identified during a previous modified Delphi study to improve their pediatric acute care knowledge and skills.MethodsAll 30 pediatric EM topics and 19/26 procedures previously identified as "must be taught by simulation" to EM residents were mapped to 15 simulation case topics. Twenty-seven authors from 16 institutions created cases and supporting materials. Each case was iteratively implemented during a peer review process at two to five sites with EM residents. Feedback from learners and facilitators was collected via electronic surveys and used to revise each case before the next implementation.ResultsThirty-five institutions participated in the peer review process. Fifty-one facilitators and 281 participants (90% EM residents) completed surveys. Most facilitators (98%) agreed or strongly agreed with the statement "This simulation case is relevant to the field of emergency medicine." A majority of facilitators and participants agreed or strongly agreed with the statements "The simulation case was realistic" (98% of facilitators, 94% of participants) and "This simulation case was effective in teaching resuscitation skills" (92% of facilitators, 98% of participants). Most participants reported confidence in knowledge and skills addressed in the learning objectives after participation.ConclusionsFacilitators and EM residents found cases from a novel simulation-based curriculum covering critical pediatric EM topics relevant, realistic, and effective. This curriculum can help provide a standardized, uniform experience for EM residents who will care for critically ill pediatric patients in their communities.

Dataset Information

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring.

Background

Methods

Results

Publications

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets