Dataset Information

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail.

ABSTRACT: INTRODUCTION:Interest is growing in specialty-specific assessments of student candidates based on clinical clerkship performance to assist in the selection process for postgraduate training. The most established and extensively used is the emergency medicine (EM) Standardized Letter of Evaluation (SLOE), serving as a substitute for the letter of recommendation. Typically developed by a program's leadership, the group SLOE strives to provide a unified institutional perspective on performance. The group SLOE lacks guidelines to direct its development raising questions regarding the assessments, processes, and standardization programs employ. This study surveys EM programs to gather validity evidence regarding the inputs and processes involved in developing group SLOEs. METHODS:A structured telephone interview was administered to assess the input data and processes employed by United States EM programs when generating group SLOEs. RESULTS:With 156/178 (87.6%) of Accreditation Council of Graduate Medical Education-approved programs responding, 146 (93.6%) reported developing group SLOEs. Issues identified in development include the following: (1) 84.9% (124/146) of programs limit the consensus process by not employing rigorous methodology; (2) several stakeholder groups (nurses, patients) do not participate in candidate assessment placing final decisions at risk for construct under-representation; and (3) clinical shift assessments don't reflect the task-specific expertise of each stakeholder group nor has the validity of each been assessed. CONCLUSION:Success of the group SLOE in its role as a summative workplace-based assessment is dependent upon valid input data and appropriate processes. This study of current program practices provides specific recommendations that would strengthen the validity arguments for the group SLOE.

SUBMITTER: Love JN

PROVIDER: S-EPMC7234706 | biostudies-literature | 2020 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail.

Love Jeffrey N JN Doty Christopher I CI Smith Jessica L JL Deiorio Nicole M NM Jordan Jaime J Van Meter Michael W MW Edens Mary Ann MA Hegarty Cullen B CB

The western journal of emergency medicine 20200421 3

<h4>Introduction</h4>Interest is growing in specialty-specific assessments of student candidates based on clinical clerkship performance to assist in the selection process for postgraduate training. The most established and extensively used is the emergency medicine (EM) Standardized Letter of Evaluation (SLOE), serving as a substitute for the letter of recommendation. Typically developed by a program's leadership, the group SLOE strives to provide a unified institutional perspective on performa ...[more]

PMID: 32421507

Similar Datasets

Project description:ObjectivesThe standardized letter of evaluation (SLOE) in emergency medicine (EM) is a widely used metric for determining interview invitations and ranking of candidates. Previous research has questioned the validity of certain sections of the SLOE. However, there remains a paucity of literature on the qualifications for EM section, which evaluates seven attributes of applicants. The aim of this study was to determine the correlation between the qualifications questions and grades, global assessment, and anticipated rank list position for EM applicants.MethodsA multi-institutional cross-sectional study was performed using SLOEs from applicants to three geographically distinct U.S. EM residency programs during the 2019-2020 application cycle. We abstracted EM rotation grade, qualifications scores, global assessment, and anticipated rank list position from the SLOEs. A Spearman correlation was calculated between each of the qualifications scores and the applicant's grades, global assessment, and anticipated rank list position in a pairwise fashion.ResultsIn total, 2,106 unique applicants (4,939 SLOEs) were included. Of the seven qualifications for EM questions, three were moderately to strongly correlated with global assessment and anticipated rank list position: "ability to develop and justify an appropriate differential and a cohesive treatment plan" (ρ = 0.65 and ρ = 0.63, respectively; p < 0.001), "how much guidance do you predict this applicant will need during residency?" (ρ = 0.68 and ρ = 0.68, respectively; p < 0.001), and "what is your prediction of success for the applicant?" (ρ = 0.69 and ρ = 0.69, respectively; p < 0.001). There was no strong correlation between the seven qualifications and grades.ConclusionsThere was a moderate to strong correlation between three of seven qualifications for EM questions (ability to develop and justify a differential and develop a cohesive plan, anticipated need for the amount of guidance, and prediction of success) with both global assessment and anticipated rank list position, suggesting that these qualifications may provide the most useful data to residency selection while some of the other factors may not be needed.

Project description:BackgroundThe majority of children seeking care in emergency departments are seen by general emergency medicine (EM) residency program graduates. Throughout training, EM residents manage fewer critically ill pediatric patients compared to adults, and the exposure to children with illness and injury requiring emergent assessment and management is often limited and sporadic across training sites. This report describes the creation of a robust set of simulation cases for EM trainees incorporating topics identified during a previous modified Delphi study to improve their pediatric acute care knowledge and skills.MethodsAll 30 pediatric EM topics and 19/26 procedures previously identified as "must be taught by simulation" to EM residents were mapped to 15 simulation case topics. Twenty-seven authors from 16 institutions created cases and supporting materials. Each case was iteratively implemented during a peer review process at two to five sites with EM residents. Feedback from learners and facilitators was collected via electronic surveys and used to revise each case before the next implementation.ResultsThirty-five institutions participated in the peer review process. Fifty-one facilitators and 281 participants (90% EM residents) completed surveys. Most facilitators (98%) agreed or strongly agreed with the statement "This simulation case is relevant to the field of emergency medicine." A majority of facilitators and participants agreed or strongly agreed with the statements "The simulation case was realistic" (98% of facilitators, 94% of participants) and "This simulation case was effective in teaching resuscitation skills" (92% of facilitators, 98% of participants). Most participants reported confidence in knowledge and skills addressed in the learning objectives after participation.ConclusionsFacilitators and EM residents found cases from a novel simulation-based curriculum covering critical pediatric EM topics relevant, realistic, and effective. This curriculum can help provide a standardized, uniform experience for EM residents who will care for critically ill pediatric patients in their communities.

Project description:INTRODUCTION:Prior research demonstrates gender differences in language used in letters of recommendation. The emergency medicine (EM) Standardized Letter of Evaluation (SLOE) format limits word count and provides detailed instructions for writers. The objective of this study is to examine differences in language used to describe men and women applicants within the SLOE narrative. METHODS:All applicants to a four-year academic EM residency program within a single application year with a first rotation SLOE available were included in the sample. We used the Linguistic Inquiry and Word Count (LIWC) program to analyze word frequency within 16 categories. Descriptive statistics, chi-squared, and t-tests were used to describe the sample; gender differences in word frequency were tested for using Mann-Whitney U tests. RESULTS:Of 1117 applicants to the residency program, 822 (82%) first-rotation SLOEs were available; 64% were men, and 36% were women. We did not find a difference in baseline characteristics including age (mean 27 years), top 25 schools (22.5%), Alpha Omega Alpha Honor Medical Society rates (13%), and having earned advanced degrees (10%). The median word count per SLOE narrative for men was 171 and for women was 180 (p = 0.15). After adjusting for letter length, word frequency differences between genders were only present in two categories: social words (women: 23 words/letter; men: 21 words/letter, p = 0.02) and ability words (women: 2 words/letter; men: 1 word/letter, p = 0.04). We were unable to detect a statistical difference between men and women applicants in the remaining categories, including words representing communal traits, agentic traits, standout adjectives, grindstone traits, teaching words, and research words. CONCLUSION:The small wording differences between genders noted in two categories were statistically significant, but of unclear real-world significance. Future work is planned to evaluate how the SLOE format may contribute to this relative lack of bias compared to other fields and formats.

Project description:ImportanceGender disparities exist throughout medicine. Recent studies have highlighted an attainment gap between male and female residents in performance evaluations on Accreditation Council for Graduate Medical Education (ACGME) milestones. Because of difficulties in blinding evaluators to gender, it remains unclear whether these observed disparities are because of implicit bias or other causes.ObjectiveTo estimate the magnitude of implicit gender bias in assessments of procedural competency in emergency medicine residents and whether the gender of the evaluator is associated with identified implicit gender bias.Design, setting, and participantsA cross-sectional study was performed from 2018 to 2020 in which emergency medicine residency faculty assessed procedural competency by evaluating videos of residents performing 3 procedures in a simulated environment. They were blinded to the intent of the study. Proceduralists were filmed performing each procedure from 2 different viewpoints simultaneously by 2 different cameras. One was a gender-blinded (ie, hands-only) view, and the other a wide-angled gender-evident (ie, whole-body) view. The faculty evaluators viewed videos in a random order and assessed procedural competency on a global rating scale with extensive validity evidence for the evaluation of video-recorded procedural performance.Main outcomes and measuresThe primary outcome was to determine if there was a difference in the evaluation of procedural competency based on gender. The secondary outcome was to determine if there was a difference in the evaluations based on the gender of the evaluator.ResultsFifty-one faculty evaluators enrolled from 19 states, with 22 male participants (43.1%), 29 female participants (56.9%), and a mean (SD) age of 37 (6.4) years. Each evaluator assessed all 60 procedures: 30 gender-blinded (hands-only view) videos and 30 identical gender-evident (wide angle) videos. There were no statistically significant differences in the study evaluators' scores of the proceduralists based on their gender, and the gender of the evaluator was not associated with the difference in mean scores.Conclusions and relevanceIn this study, we did not identify a difference in the evaluation of procedural competency based upon the gender of the resident proceduralist or the gender of the faculty evaluator.

Dataset Information

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail.

Publications

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets