Dataset Information

Milestone Implementation's Impact on Narrative Comments and Perception of Feedback for Internal Medicine Residents: a Mixed Methods Study.

ABSTRACT:

Background

Feedback is a critical element of graduate medical education. Narrative comments on evaluation forms are a source of feedback for residents. As a shared mental model for performance, milestone-based evaluations may impact narrative comments and resident perception of feedback.

Objective

To determine if milestone-based evaluations impacted the quality of faculty members' narrative comments on evaluations and, as an extension, residents' perception of feedback.

Design

Concurrent mixed methods study, including qualitative analysis of narrative comments and survey of resident perception of feedback.

Participants

Seventy internal medicine residents and their faculty evaluators at the University of Utah.

Approach

Faculty narrative comments from 248 evaluations pre- and post-milestone implementation were analyzed for quality and Accreditation Council for Graduate Medical Education competency by area of strength and area for improvement. Seventy residents were surveyed regarding quality of feedback pre- and post-milestone implementation.

Key results

Qualitative analysis of narrative comments revealed nearly all evaluations pre- and post-milestone implementation included comments about areas of strength but were frequently vague and not related to competencies. Few evaluations included narrative comments on areas for improvement, but these were of higher quality compared to areas of strength (p < 0.001). Overall resident perception of quality of narrative comments was low and did not change following milestone implementation (p?=?0.562) for the 86% of residents (N?=?60/70) who completed the pre- and post-surveys.

Conclusions

The quality of narrative comments was poor, and there was no evidence of improved quality following introduction of milestone-based evaluations. Comments on areas for improvement were of higher quality than areas of strength, suggesting an area for targeted intervention. Residents' perception of feedback quality did not change following implementation of milestone-based evaluations, suggesting that in the post-milestone era, internal medicine educators need to utilize additional interventions to improve quality of feedback.

SUBMITTER: Raaum SE

PROVIDER: S-EPMC6544770 | biostudies-literature | 2019 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Milestone Implementation's Impact on Narrative Comments and Perception of Feedback for Internal Medicine Residents: a Mixed Methods Study.

Raaum Sonja E SE Lappe Katie K Colbert-Getz Jorie M JM Milne Caroline K CK

Journal of general internal medicine 20190601 6

<h4>Background</h4>Feedback is a critical element of graduate medical education. Narrative comments on evaluation forms are a source of feedback for residents. As a shared mental model for performance, milestone-based evaluations may impact narrative comments and resident perception of feedback.<h4>Objective</h4>To determine if milestone-based evaluations impacted the quality of faculty members' narrative comments on evaluations and, as an extension, residents' perception of feedback.<h4>Design< ...[more]

PMID: 30891692

Similar Datasets

Project description:BACKGROUND:Feedback is essential for improving the skills of continuing medical education (CME) presenters. However, there has been little research on improving the quality of feedback to CME presenters. OBJECTIVES:To validate an instrument for generating balanced and behavior-specific feedback from a national cross-section of participants to presenters at a large internal medicine CME course. DESIGN, SETTING, AND PARTICIPANTS:A prospective, randomized validation study with qualitative data analysis that included all 317 participants at a Mayo Clinic internal medicine CME course in 2009. MEASUREMENTS:An 8-item (5-point Likert scales) CME faculty assessment enhanced study form (ESF) was designed based on literature and expert review. Course participants were randomized to a standard form, a generic study form (GSF), or the ESF. The dimensionality of instrument scores was determined using factor analysis to account for clustered data. Internal consistency and interrater reliabilities were calculated. Associations between overall feedback scores and presenter and presentation variables were identified using generalized estimating equations to account for multiple observations within talk and speaker combinations. Two raters reached consensus on qualitative themes and independently analyzed narrative entries for evidence of balanced and behavior-specific comments. RESULTS:Factor analysis of 5,241 evaluations revealed a uni-dimensional model for measuring CME presenter feedback. Overall internal consistency (Cronbach alpha?=?0.94) and internal consistency reliability (ICC range 0.88-0.95) were excellent. Feedback scores were associated with presenters' academic ranks (mean score): Instructor (4.12), Assistant Professor (4.38), Associate Professor (4.56), Professor (4.70) (p?=?0.046). Qualitative analysis revealed that the ESF generated the highest numbers of balanced comments (GSF?=?11, ESF?=?26; p?=?0.01) and behavior-specific comments (GSF?=?64, ESF?=?104; p?=?0.001). CONCLUSIONS:We describe a practical and validated method for generating balanced and behavior-specific feedback for CME presenters in internal medicine. Our simple method for prompting course participants to give balanced and behavior-specific comments may ultimately provide CME presenters with feedback for improving their presentations.

Project description:BackgroundNew virtual resources ("novel resources") have been incorporated into medical education. No recent large studies about their use and perception among internal medicine (IM) residents exist.ObjectiveCharacterize the use and perceived helpfulness of educational resources.DesignNationwide survey from December 2019 to March 2020.ParticipantsIM residents in the USA.Main measuresResidents were surveyed on their use and their perceived helpfulness of resources for both attaining general medical knowledge and for point-of-care (POC) learning. Traditional resources included board review resources, clinical experience, digital clinical resources (e.g., UpToDate), journal articles, pocket references, professional guidelines, textbooks, and residency curricula. Novel resources included Twitter, video streaming platforms (e.g., YouTube), online blogs, podcasts, and Wikipedia.Key resultsWe had 662 respondents from 55 residency programs across 26 states. On average, residents used 9 total resources (7 traditional and 2 novel). Digital clinical resources and clinical experience were used by all residents and found helpful by the highest percentage of residents (96% and 94%, respectively). Journal articles were next (used by 90%), followed by board review resources and residency curricula (both used by 85%). Their perceived helpfulness varied, from 90% for board review resources, to 66% for journal articles and 64% for residency curricula, the lowest perceived helpfulness of any traditional resource. Podcasts and video streaming platforms were used as frequently as textbooks (58-59%), but were rated as helpful more frequently (75% and 82% vs 66%, respectively).ConclusionsDigital clinical resources, video streaming platforms, and podcasts were perceived as helpful, underscoring the importance of ensuring their integration into medical education to complement clinical experience and other traditional resources which remain highly valued by residents.ImportanceOur findings can inform residency programs as they transition to virtual curricula in the wake of the COVID-19 pandemic.

Project description:INTRODUCTION:Evaluators use assessment data to make judgments on resident performance within the Accreditation Council for Graduate Medical Education (ACGME) milestones framework. While workplace-based narrative assessments (WBNA) offer advantages to rating scales, validity evidence for their use in assessing the milestone sub-competencies is lacking. This study aimed to determine the frequency of sub-competencies assessed through WBNAs in an emergency medicine (EM) residency program. METHODS:We performed a retrospective analysis of WBNAs of postgraduate year (PGY) 2-4 residents. A shared mental model was established by reading and discussing the milestones framework, and we created a guide for coding WBNAs to the milestone sub-competencies in an iterative process. Once inter-rater reliability was satisfactory, raters coded each WBNA to the 23 EM milestone sub-competencies. RESULTS:We analyzed 2517 WBNAs. An average of 2.04 sub-competencies were assessed per WBNA. The sub-competencies most frequently identified were multitasking, medical knowledge, practice-based performance improvement, patient-centered communication, and team management. The sub-competencies least frequently identified were pharmacotherapy, airway management, anesthesia and acute pain management, goal-directed focused ultrasound, wound management, and vascular access. Overall, the frequency with which WBNAs assessed individual sub-competencies was low, with 14 of the 23 sub-competencies being assessed in less than 5% of WBNAs. CONCLUSION:WBNAs identify few milestone sub-competencies. Faculty assessed similar sub-competencies related to interpersonal and communication skills, practice-based learning and improvement, and medical knowledge, while neglecting sub-competencies related to patient care and procedural skills. These findings can help shape faculty development programs designed to improve assessments of specific workplace behaviors and provide more robust data for the summative assessment of residents.

Dataset Information

Milestone Implementation's Impact on Narrative Comments and Perception of Feedback for Internal Medicine Residents: a Mixed Methods Study.

Background

Objective

Design

Participants

Approach

Key results

Conclusions

Publications

Milestone Implementation's Impact on Narrative Comments and Perception of Feedback for Internal Medicine Residents: a Mixed Methods Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets