Dataset Information

Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?

ABSTRACT: INTRODUCTION:Evaluators use assessment data to make judgments on resident performance within the Accreditation Council for Graduate Medical Education (ACGME) milestones framework. While workplace-based narrative assessments (WBNA) offer advantages to rating scales, validity evidence for their use in assessing the milestone sub-competencies is lacking. This study aimed to determine the frequency of sub-competencies assessed through WBNAs in an emergency medicine (EM) residency program. METHODS:We performed a retrospective analysis of WBNAs of postgraduate year (PGY) 2-4 residents. A shared mental model was established by reading and discussing the milestones framework, and we created a guide for coding WBNAs to the milestone sub-competencies in an iterative process. Once inter-rater reliability was satisfactory, raters coded each WBNA to the 23 EM milestone sub-competencies. RESULTS:We analyzed 2517 WBNAs. An average of 2.04 sub-competencies were assessed per WBNA. The sub-competencies most frequently identified were multitasking, medical knowledge, practice-based performance improvement, patient-centered communication, and team management. The sub-competencies least frequently identified were pharmacotherapy, airway management, anesthesia and acute pain management, goal-directed focused ultrasound, wound management, and vascular access. Overall, the frequency with which WBNAs assessed individual sub-competencies was low, with 14 of the 23 sub-competencies being assessed in less than 5% of WBNAs. CONCLUSION:WBNAs identify few milestone sub-competencies. Faculty assessed similar sub-competencies related to interpersonal and communication skills, practice-based learning and improvement, and medical knowledge, while neglecting sub-competencies related to patient care and procedural skills. These findings can help shape faculty development programs designed to improve assessments of specific workplace behaviors and provide more robust data for the summative assessment of residents.

SUBMITTER: Diller D

PROVIDER: S-EPMC6948702 | biostudies-literature | 2019 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?

Diller David D Cooper Shannon S Jain Aarti A Lam Chun Nok CN Riddell Jeff J

The western journal of emergency medicine 20191220 1

<h4>Introduction</h4>Evaluators use assessment data to make judgments on resident performance within the Accreditation Council for Graduate Medical Education (ACGME) milestones framework. While workplace-based narrative assessments (WBNA) offer advantages to rating scales, validity evidence for their use in assessing the milestone sub-competencies is lacking. This study aimed to determine the frequency of sub-competencies assessed through WBNAs in an emergency medicine (EM) residency program.<h4 ...[more]

PMID: 31913841

Similar Datasets

Project description:ImportanceAlthough implicit bias in medical training has long been suspected, it has been difficult to study using objective measures, and the influence of sex and gender in the evaluation of medical trainees is unknown. The emergency medicine (EM) milestones provide a standardized framework for longitudinal resident assessment, allowing for analysis of resident performance across all years and programs at a scope and level of detail never previously possible.ObjectiveTo compare faculty-observed training milestone attainment of male vs female residency training.Design, setting, and participantsThis multicenter, longitudinal, retrospective cohort study took place at 8 community and academic EM training programs across the United States from July 1, 2013, to July 1, 2015, using a real-time, mobile-based, direct-observation evaluation tool. The study examined 33 456 direct-observation subcompetency evaluations of 359 EM residents by 285 faculty members.Main outcomes and measuresMilestone attainment for male and female EM residents as observed by male and female faculty throughout residency and analyzed using multilevel mixed-effects linear regression modeling.ResultsA total of 33 456 direct-observation evaluations were collected from 359 EM residents (237 men [66.0%] and 122 women [34.0%]) by 285 faculty members (194 men [68.1%] and 91 women [31.9%]) during the study period. Female and male residents achieved similar milestone levels during the first year of residency. However, the rate of milestone attainment was 12.7% (0.07 levels per year) higher for male residents through all of residency (95% CI, 0.04-0.09). By graduation, men scored approximately 0.15 milestone levels higher than women, which is equivalent to 3 to 4 months of additional training, given that the average resident gains approximately 0.52 levels per year using our model (95% CI, 0.49-0.54). No statistically significant differences in scores were found based on faculty evaluator gender (effect size difference, 0.02 milestone levels; 95% CI for males, -0.09 to 0.11) or evaluator-evaluatee gender pairing (effect size difference, -0.02 milestone levels; 95% CI for interaction, -0.05 to 0.01).Conclusions and relevanceAlthough male and female residents receive similar evaluations at the beginning of residency, the rate of milestone attainment throughout training was higher for male than female residents across all EM subcompetencies, leading to a gender gap in evaluations that continues until graduation. Faculty should be cognizant of possible gender bias when evaluating medical trainees.

Project description:BackgroundFeedback is a critical element of graduate medical education. Narrative comments on evaluation forms are a source of feedback for residents. As a shared mental model for performance, milestone-based evaluations may impact narrative comments and resident perception of feedback.ObjectiveTo determine if milestone-based evaluations impacted the quality of faculty members' narrative comments on evaluations and, as an extension, residents' perception of feedback.DesignConcurrent mixed methods study, including qualitative analysis of narrative comments and survey of resident perception of feedback.ParticipantsSeventy internal medicine residents and their faculty evaluators at the University of Utah.ApproachFaculty narrative comments from 248 evaluations pre- and post-milestone implementation were analyzed for quality and Accreditation Council for Graduate Medical Education competency by area of strength and area for improvement. Seventy residents were surveyed regarding quality of feedback pre- and post-milestone implementation.Key resultsQualitative analysis of narrative comments revealed nearly all evaluations pre- and post-milestone implementation included comments about areas of strength but were frequently vague and not related to competencies. Few evaluations included narrative comments on areas for improvement, but these were of higher quality compared to areas of strength (p < 0.001). Overall resident perception of quality of narrative comments was low and did not change following milestone implementation (p = 0.562) for the 86% of residents (N = 60/70) who completed the pre- and post-surveys.ConclusionsThe quality of narrative comments was poor, and there was no evidence of improved quality following introduction of milestone-based evaluations. Comments on areas for improvement were of higher quality than areas of strength, suggesting an area for targeted intervention. Residents' perception of feedback quality did not change following implementation of milestone-based evaluations, suggesting that in the post-milestone era, internal medicine educators need to utilize additional interventions to improve quality of feedback.

Project description:BackgroundThe field of clinical informatics (CI), and specifically the electronic health record, has been identified as a key facilitator to achieve a sustainable evidence-based health care system for the future. International graduate medical education (GME) programs have been challenged to ensure that their trainees are provided with appropriate skills to deliver effective and efficient health care in an evolving environment.ObjectivesThis study explored how international emergency medicine (EM) specialist training standards address competencies and training in relevant areas of CI.MethodsA list of categories of CI competencies relative to EM was developed following a thematic review of published references documenting CI curriculum and competencies. Publicly available documents outlining core content, curriculum, and competencies from international organizations responsible for specialty GME and/or credentialing in EM for Australasia, Canada, Europe, the United Kingdom, and the United States were identified. These EM training standards were reviewed to identify inclusion of topics related to the relevant categories of CI competencies.ResultsA total of 23 EM curriculum documents were included in the review. Curricula content related to critical appraisal/evidence-based medicine, leadership, quality improvement, and privacy/security were included in all EM curricula. The CI topics related to fundamental computer skills, computerized provider order entry, and patient-centered informatics were only included in the EM curricula documents for the United States and were absent for the other jurisdictions.ConclusionThere is variation in the CI-related content of the international EM specialty training standards reviewed. Given the increasing importance of CI in the future delivery of health care, organizations responsible for training and credentialing specialist emergency physicians must ensure that their training standards incorporate relevant CI content, thus ensuring that their trainees gain competence in essential aspects of CI.

Dataset Information

Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?

Publications

Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets