Dataset Information

Developing and Implementing a Multisource Feedback Tool to Assess Competencies of Emergency Medicine Residents in the United States.

ABSTRACT:

Objectives

Multisource feedback (MSF) has potential value in learner assessment, but has not been broadly implemented nor studied in emergency medicine (EM). This study aimed to adapt existing MSF instruments for emergency department implementation, measure feasibility, and collect initial validity evidence to support score interpretation for learner assessment.

Methods

Residents from eight U.S. EM residency programs completed a self-assessment and were assessed by eight physicians, eight nonphysician colleagues, and 25 patients using unique instruments. Instruments included a five-point rating scale to assess interpersonal and communication skills, professionalism, systems-based practice, practice-based learning and improvement, and patient care. MSF feasibility was measured by percentage of residents who collected the target number of instruments. To develop internal structure validity evidence, Cronbach's alpha was calculated as a measure of internal consistency.

Results

A total of 125 residents collected a mean of 7.0 physician assessments (n = 752), 6.7 nonphysician assessments (n = 775), and 17.8 patient assessments (n = 2,100) with respective response rates of 67.2, 75.2, and 77.5%. Cronbach's alpha values for physicians, nonphysicians, patients, and self were 0.97, 0.97, 0.96, and 0.96, respectively.

Conclusions

This study demonstrated that MSF implementation is feasible, although challenging. The tool and its scale demonstrated excellent internal consistency. EM educators may find the adaptation process and tools applicable to their learners.

SUBMITTER: LaMantia J

PROVIDER: S-EPMC6001492 | biostudies-literature | 2017 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Developing and Implementing a Multisource Feedback Tool to Assess Competencies of Emergency Medicine Residents in the United States.

LaMantia Joseph J Yarris Lalena M LM Sunga Kharmene K Weizberg Moshe M Hart Danielle D Farina Gino G Rodriguez Elliot E Lucas Raymond R Mahmooth Zayan Z Snock Alexandra A Lockyear Jocelyn J

AEM education and training 20170615 3

<h4>Objectives</h4>Multisource feedback (MSF) has potential value in learner assessment, but has not been broadly implemented nor studied in emergency medicine (EM). This study aimed to adapt existing MSF instruments for emergency department implementation, measure feasibility, and collect initial validity evidence to support score interpretation for learner assessment.<h4>Methods</h4>Residents from eight U.S. EM residency programs completed a self-assessment and were assessed by eight physician ...[more]

PMID: 30051042

Similar Datasets

Project description:ObjectivesResidents in emergency medicine have reported dissatisfaction with feedback. One strategy to improve feedback is to enhance learners' feedback literacy-i.e., capabilities as seekers, processors, and users of performance information. To do this, however, the context in which feedback occurs needs to be understood. We investigated how residents typically engage with feedback in an emergency department, along with the potential opportunities to improve feedback engagement in this context. We used this information to develop a program to improve learners' feedback literacy in context and traced the reported translation to practice.MethodsWe conducted a year-long design-based research study informed by agentic feedback principles. Over five cycles in 2019, we interviewed residents and iteratively developed a feedback literacy program. Sixty-six residents participated and data collected included qualitative evaluation surveys (n = 55), educator-written reflections (n = 5), and semistructured interviews with residents (n = 21). Qualitative data were analyzed using framework analysis.ResultsWhen adopting an agentic stance, residents reported changes to the frequency and tenor of their feedback conversations, rendering the interactions more helpful. Despite reporting overall shifts in their conceptions of feedback, they needed to adjust their feedback engagement depending on changing contextual factors such as workload. These microsocial adjustments suggest their feedback literacy develops through an interdependent process of individual intention for feedback engagement-informed by an agentic stance-and dynamic adjustment in response to the environment.ConclusionsResident feedback literacy is profoundly contextualized, so developing feedback literacy in emergency contexts is more nuanced than previously reported. While feedback literacy can be supported through targeted education, our findings raise questions for understanding how emergency medicine environments afford and constrain learner feedback engagement. Our findings also challenge the extent to which this contextual feedback know-how can be "developed" purposefully outside of the everyday work.

Project description:IntroductionThe Accreditation Council for Graduate Medical Education (ACGME) specifically notes multisource feedback (MSF) as a recommended means of resident assessment in the emergency medicine (EM) Milestones. High-fidelity simulation is an environment wherein residents can receive MSF from various types of healthcare professionals. Previously, the Queen's Simulation Assessment Tool (QSAT) has been validated for faculty to assess residents in five categories: assessment; diagnostic actions; therapeutic actions; interpersonal communication, and overall assessment. We sought to determine whether the QSAT could be used to provide MSF using a standardized simulation case.MethodsProspectively after institutional review board approval, residents from a dual ACGME/osteopathic-approved postgraduate years (PGY) 1-4 EM residency were consented for participation. We developed a standardized resuscitation after overdose case with specific 1-5 Likert anchors used by the QSAT. A PGY 2-4 resident participated in the role of team leader, who completed a QSAT as self-assessment. The team consisted of a PGY-1 peer, an emergency medical services (EMS) provider, and a nurse. Two core faculty were present to administer the simulation case and assess. Demographics were gathered from all participants completing QSATs. We analyzed QSATs by each category and on cumulative score. Hypothesis testing was performed using intraclass correlation coefficients (ICC), with 95% confidence intervals. Interpretation of ICC results was based on previously published definitions.ResultsWe enrolled 34 team leader residents along with 34 nurses. A single PGY-1, a single EMS provider and two faculty were also enrolled. Faculty provided higher cumulative QSAT scores than the other sources of MSF. QSAT scores did not increase with team leader PGY level. ICC for inter-rater reliability for all sources of MSF was 0.754 (0.572-0.867). Removing the self-evaluation scores increased inter-rater reliability to 0.838 (0.733-0.910). There was lesser agreement between faculty and nurse evaluations than from the EMS or peer evaluation.ConclusionIn this single-site cohort using an internally developed simulation case, the QSAT provided MSF with excellent reliability. Self-assessment decreases the reliability of the MSF, and our data suggest self-assessment should not be a component of MSF. Use of the QSAT for MSF may be considered as a source of data for clinical competency committees.

Project description:BACKGROUND:Multisource feedback (MSF) is increasingly being used to assess trainee performance, with different assessor groups fulfilling a crucial role in utility of assessment data. However, in health professions education, research on assessor behaviors in MSF is limited. When assessing trainee performance in work settings, assessors use multidimensional conceptualizations of what constitutes effective performance, also called personal performance theories, to distinguish between various behaviors and sub competencies., This may not only explain assessor variability in Multi Source Feedback, but also result in differing acceptance (and use) of assessment data for developmental purposes. The purpose of this study was to explore performance theories of various assessor groups (residents and nurses) when assessing performance of residents. METHODS:A constructivist, inductive qualitative research approach and semi-structured interviews following MSF were used to explore performance theories of 14 nurses and 15 residents in the department of internal medicine at Aga Khan University (AKU). Inductive thematic content analysis of interview transcripts was used to identify and compare key dimensions in residents' and nurses' performance theories used in evaluation of resident performance. RESULTS:Seven major themes, reflecting key dimensions of assessors' performance theories, emerged from the qualitative data, namely; communication skills, patient care, accessibility, teamwork skills, responsibility, medical knowledge and professional attitude. There were considerable overlaps, but also meaningful differences in the performance theories of residents and the nurses, especially with respect to accessibility, teamwork and medical knowledge. CONCLUSION:Residents' and nurses' performance theories for assessing resident performance overlap to some extent, yet also show meaningful differences with respect to the performance dimensions they pay attention to or consider most important. In MSF, different assessor groups may therefore hold different performance theories, depending on their role. Our results further our understanding of assessor source effects in MSF. Implications of our findings are related to implementation of MSF, design of rating scales as well as interpretation and use of MSF data for selection and performance improvement.

Dataset Information

Developing and Implementing a Multisource Feedback Tool to Assess Competencies of Emergency Medicine Residents in the United States.

Objectives

Methods

Results

Conclusions

Publications

Developing and Implementing a Multisource Feedback Tool to Assess Competencies of Emergency Medicine Residents in the United States.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets