Dataset Information

Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: an observational study.

ABSTRACT: BACKGROUND: Research has shown that narrative feedback, (self) reflections and a plan to undertake and evaluate improvements are key factors for effective feedback on clinical performance. We investigated the quantity of narrative comments comprising feedback (by trainers), self-reflections (by trainees) and action plans (by trainer and trainee) entered on a mini-CEX form that was modified for use in general practice training and to encourage trainers and trainees to provide narrative comments. In view of the importance of specificity as an indicator of feedback quality, we additionally examined the specificity of the comments. METHOD: We collected and analysed modified mini-CEX forms completed by GP trainers and trainees. Since each trainee has the same trainer for the duration of one year, we used trainer-trainee pairs as the unit of analysis. We determined for all forms the frequency of the different types of narrative comments and rated their specificity on a three-point scale: specific, moderately specific, not specific. Specificity was compared between trainee-trainer pairs. RESULTS: We collected 485 completed modified mini-CEX forms from 54 trainees (mean of 8.8 forms per trainee; range 1-23; SD 5.6). Trainer feedback was more frequently provided than trainee self-reflections, and action plans were very rare. The comments were generally specific, but showed large differences between trainee-trainer pairs. CONCLUSION: The frequency of self-reflection and action plans varied, all comments were generally specific and there were substantial and consistent differences between trainee-trainer pairs in the specificity of comments. We therefore conclude that feedback is not so much determined by the instrument as by the users. Interventions to improve the educational effects of the feedback procedure should therefore focus more on the users than on the instruments.

SUBMITTER: Pelgrim EA

PROVIDER: S-EPMC3505168 | biostudies-literature | 2012

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: an observational study.

Pelgrim Elisabeth A M EA Kramer Anneke W M AW Mokkink Henk G A HG Van der Vleuten Cees P M CP

BMC medical education 20121018

<h4>Background</h4>Research has shown that narrative feedback, (self) reflections and a plan to undertake and evaluate improvements are key factors for effective feedback on clinical performance. We investigated the quantity of narrative comments comprising feedback (by trainers), self-reflections (by trainees) and action plans (by trainer and trainee) entered on a mini-CEX form that was modified for use in general practice training and to encourage trainers and trainees to provide narrative com ...[more]

PMID: 23075126

Similar Datasets

Project description:ProblemWritten feedback is often overly positive, nonspecific, and difficult to interpret. Learner satisfaction with written feedback is low and obtaining written feedback that encourages self-reflection is challenging. Improving feedback quality is laborious and only modestly effective.ApproachThe authors developed the LEAF (Learner-Engaged Analysis of Feedback) method to improve learner satisfaction with, and reflection on, existing written feedback. The method pairs a learner and coach to methodically identify themes in the learner's written feedback. Themes occurring more frequently or less frequently than typical offer areas for reflection, as they may identify learners' relative strengths or weaknesses. The method was introduced at the Massachusetts General Hospital in 2017 during program director (PD) meetings with anesthesiology residents. In 2018, resident satisfaction was measured (1 to 5 Likert-type questions, 1 = "not at all satisfied," 5 = "extremely satisfied") for 4 feedback sources, 2 related to the LEAF method (PD meetings, written feedback) and 2 unrelated (verbal feedback, mentor feedback). Residents' comments were qualitatively assessed to explore the impact on self-reflection.OutcomesResidents who had participated in a LEAF session (n = 54), compared with those who had not (n = 11), reported higher satisfaction with written feedback (mean 3.1 versus 2.5, d = 0.53, P = .03) and PD meeting feedback (mean 3.8 versus 2.8, d = 0.80, P = .03). There were no significant differences between groups for satisfaction with feedback unrelated to the LEAF method. Qualitative analysis of comments suggested that residents found the method useful for providing holistic self-assessment, facilitating goal setting, uncovering blind spots, and improving feedback interpretation.Next stepsNext steps should include studies determining if the association between increased learner satisfaction with written feedback and the LEAF method is causal, and whether this feedback process changes learners' subsequent behaviors.

Project description:ObjectivesThe Serious Illness Conversation Guide (SICG) has emerged as a framework for conversations with patients with a serious illness diagnosis. This study reports on narratives generated from open-ended questions of a novel assessment tool, the Serious Illness Conversation-Evaluation Exercise (SIC-Ex), to assess resident-led conversations with patients in oncology outpatient clinics.DesignQualitative study using template analysis.SettingThree academic cancer centres in Canada.Participants7 resident physicians (trainees), 7 patients from outpatient cancer clinics, 10 preceptors (raters) consisting of medical oncologists, palliative care physicians and radiation oncologists.InterventionsEach trainee conducted an SIC with a patient, which was videotaped. The raters watched the videos and evaluated each trainee using the novel SIC-Ex and the reference Calgary-Cambridge Guide (CCG) initially and again 3 months later. Two independent coders used template analysis to code the raters' narrative comments and identify themes/subthemes.Outcome measuresHow narrative comments aligned with elements of the CCG and SICG.ResultsTemplate analysis yielded four themes: adhering to SICG, engaging patients and family members, conversation management and being mindful of demeanour. Narrative comments identified numerous verbal and non-verbal elements essential to SICG. Some comments addressing general skills in engaging patients/families and managing the conversation (eg, setting agenda, introduction, planning, exploring, non-verbal communication) related to both the CCG and SICG, whereas other comments such as identifying substitute decision maker(s), affirming commitment and introducing Advance Care Planning were specific to the SICG.ConclusionsNarrative comments generated by SIC-Ex provided detailed and nuanced insights into trainees' competence in SIC, beyond the numerical ratings of SIC-Ex and the general communication skills outlined in the CCG, and may contribute to a more fulsome assessment of SIC skills.

Project description:BackgroundCultural differences might challenge the acceptance of the implementation of assessment formats that are developed in other countries. Acceptance of assessment formats is essential for its effectiveness; therefore, we explored the views of students and specialists on the practicality and impact on learning of these formats. This study was conducted to explore Indonesian students' and specialists' appreciation of the implementation of the Mini-Clinical Evaluation Exercise (Mini-CEX) in Indonesian clerkships.MethodsThis study was conducted at the Universitas Gadjah Mada, Indonesia. Participants were 52 students and 21 specialists in neurology and 78 students and 50 specialists in internal medicine. They were asked to complete a 19-item questionnaire that covered the characteristics of the mini-CEX such as its practicality, and the impact on learning and professional development. We used a Mann-Whitney U test to analyse the data.ResultsIn total, 124 students (46 from neurology and 78 from internal medicine) and 38 specialists (13 from neurology and 25 from internal medicine) participated in this study. Students and specialists were positive about the practicality of the mini-CEX and the impact of this assessment format on learning and on professional development. The Mann-Whitney U test showed that there were no significant differences between students' and specialists' opinions on the mini-CEX, except for 2 items: specialists' appreciation of direct observation (mean rank = 93.16) was statistically significantly higher than students' appreciation of it (mean rank = 77.93; z = 2.065; p < 0.05), but students' appreciation of the item that students' past mini-CEX results affected their recent mini-CEX outcomes (mean rank = 85.29) was significantly higher than specialists' appreciation of it (mean rank = 69.12; z = 2140; p < 0.05).ConclusionStudents and specialists were positive about the mini-CEX in Indonesian clerkships, although it was developed and validated in another culture. We found only small differences between their appreciations, which could be explained by the patterns of specialist-student interaction in Indonesian culture as large power distance and low individualism country.

Project description:BackgroundThe Objective Structured Clinical Exam (OSCE) is a useful means of generating meaningful feedback. OSCE feedback may be in various forms (written, face to face and audio or video recordings). Studies on OSCE feedback are uncommon, especially involving Asian medical students.MethodsWe compared two methods of OSCE feedback delivered to fourth year medical students in Malaysia: (i) Face to face (FTF) immediate feedback (semester one) (ii) Individualised enhanced written (EW) feedback containing detailed scores in each domain, examiners' free text comments and the marking rubric (semester two). Both methods were evaluated by students and staff examiners, and students' responses were compared against their OSCE performance.ResultsOf the 116 students who sat for both formative OSCEs, 82.8% (n=96) and 86.2% (n=100) responded to the first and second survey respectively. Most students were comfortable to receive feedback (91.3% in FTF, 96% in EW) with EW feedback associated with higher comfort levels (p=0.022). Distress affected a small number with no differences between either method (13.5% in FTF, 10% in EW, p=0.316). Most students perceived both types of feedback improved their performance (89.6% in FTF, 95% in EW); this perception was significantly stronger for EW feedback (p=0.008). Students who preferred EW feedback had lower OSCE scores compared to those preferring FTF feedback (mean scores ± SD: 43.8 ± 5.3 in EW, 47.2 ± 6.5 in FTF, p=0.049). Students ranked the "marking rubric" to be the most valuable aspect of the EW feedback. Tutors felt both methods of feedback were equally beneficial. Few examiners felt they needed training (21.4% in FTF, 15% in EW) but students perceived this need for tutors' training differently (53.1% in FTF, 46% in EW) CONCLUSION: Whilst both methods of OSCE feedback were highly valued, students preferred to receive EW feedback and felt it was more beneficial. Learning cultures of Malaysian students may have influenced this view. Information provided in EW feedback should be tailored accordingly to provide meaningful feedback in OSCE exams.

Project description:ObjectivesTo examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.DesignQualitative text analysis supplemented by quantitative descriptive analysis.SettingInternal Medicine Departments in Zealand, Denmark.Participants48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).InterventionThis study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either 'no reasons for concern' or 'possibly some concern', thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.ResultsOut of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter-rater correlations between four authors were 0.7-1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.ConclusionsA purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees' performances based on the MSF reports.

Dataset Information

Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: an observational study.

Publications

Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: an observational study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets