Confounding factors in using upward feedback to assess the quality of medical training: a systematic review.
Ontology highlight
ABSTRACT:
SUBMITTER: Zhou AY
PROVIDER: S-EPMC4309940 | biostudies-literature | 2014
REPOSITORIES: biostudies-literature
Publications
Confounding factors in using upward feedback to assess the quality of medical training: a systematic review.
Journal of educational evaluation for health professions 20140813
<h4>Purpose</h4>Upward feedback is becoming more widely used in medical training as a means of quality control. Multiple biases exist, thus the accuracy of upward feedback is debatable. This study aims to identify factors that could influence upward feedback, especially in medical training.<h4>Methods</h4>A systematic review using a structured search strategy was performed. Thirty-five databases were searched. Results were reviewed and relevant abstracts were shortlisted. All studies in English, ...[more]
Similar Datasets
Project description:BackgroundIn this research paper we report on the quality of feedback provided in the logbooks of pre-clinical undergraduate students based on a model of 'actionable feedback'. Feedback to clinical learners about their performance is crucial to their learning, which ultimately impacts on their development into competent clinicians. Due to students' concerns regarding the inconsistency and quality of feedback provided by clinicians, a structured feedback improvement strategy to move feedback forward was added to the clinical skills logbook. The instrument was also extended for peer assessment. This study aims to assess the quality of feedback using the deliberate practice framework.MethodsA feedback scoring system was used to retrospectively assess the quality of tutor and peer logbook feedback provided to second and third year medical students to identify deliberate practice components i.e. task, performance gap and action plan. The sample consisted of 425 second year and 600 third year feedback responses over a year.ResultsAll three deliberate practice components were observed in the majority of the written feedback for both classes. The frequency was higher in peer (83%, 89%) than tutor logbook assessments (51%, 67%) in both classes respectively. Average tutor and peer task, gap and action feedback scores ranged from 1.84-2.07 and 1.93-2.21 respectively. The overall quality of feedback provided by the tutor and peer was moderate and less specific (average score < or = 2). The absence of the three components was noted in only 1% of the feedback responses in both 2nd and 3rd year.ConclusionThis study found that adding in a feed-forward strategy to the logbooks increased the overall quality of tutor and peer feedback as the task, gap and action plans were described. Deliberate practice framework provides an objective assessment of tutor and peer feedback quality and can be used for faculty development and training. The findings from our study suggest that the ratings from the tool can also be used as guidelines to provide feedback providers with feedback on the quality of feedback they provided. This includes specifically describing a task, performance gap and providing a learning plan as feed-forward to enhance feedback given.
| S-EPMC6460682 | biostudies-literature
Project description:Background?:Leadership is a critical component of physician competence, yet the best approaches for developing leadership skills for physicians in training remain undefined. Objective?:We systematically reviewed the literature on existing leadership curricula in graduate medical education (GME) to inform leadership program development. Methods?:Using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines, we searched MEDLINE, ERIC, EMBASE, and MedEdPORTAL through October 2015 using search terms to capture GME leadership curricula. Abstracts were reviewed for relevance, and included studies were retrieved for full-text analysis. Article quality was assessed using the Best Evidence in Medical Education (BEME) index. Results?:A total of 3413 articles met the search criteria, and 52 were included in the analysis. Article quality was low, with 21% (11 of 52) having a BEME score of 4 or 5. Primary care specialties were the most represented (58%, 30 of 52). The majority of programs were open to all residents (81%, 42 of 52). Projects and use of mentors or coaches were components of 46% and 48% of curricula, respectively. Only 40% (21 of 52) were longitudinal throughout training. The most frequent pedagogic methods were lectures, small group activities, and cases. Common topics included teamwork, leadership models, and change management. Evaluation focused on learner satisfaction and self-assessed knowledge. Longitudinal programs were more likely to be successful. Conclusions?:GME leadership curricula are heterogeneous and limited in effectiveness. Small group teaching, project-based learning, mentoring, and coaching were more frequently used in higher-quality studies.
| S-EPMC5901791 | biostudies-literature
Project description:BackgroundThere is significant variability in the performance and outcomes of invasive medical procedures such as percutaneous coronary intervention, endoscopy, and bronchoscopy. Peer evaluation is a common mechanism for assessment of clinician performance and care quality, and may be ideally suited for the evaluation of medical procedures. We therefore sought to perform a systematic review to identify and characterize peer evaluation tools for practicing clinicians, assess evidence supporting the validity of peer evaluation, and describe best practices of peer evaluation programs across multiple invasive medical procedures.MethodsA systematic search of Medline and Embase (through September 7, 2021) was conducted to identify studies of peer evaluation and feedback relating to procedures in the field of internal medicine and related subspecialties. The methodological quality of the studies was assessed. Data were extracted on peer evaluation methods, feedback structures, and the validity and reproducibility of peer evaluations, including inter-observer agreement and associations with other quality measures when available.ResultsOf 2,135 retrieved references, 32 studies met inclusion criteria. Of these, 21 were from the field of gastroenterology, 5 from cardiology, 3 from pulmonology, and 3 from interventional radiology. Overall, 22 studies described the development or testing of peer scoring systems and 18 reported inter-observer agreement, which was good or excellent in all but 2 studies. Only 4 studies, all from gastroenterology, tested the association of scoring systems with other quality measures, and no studies tested the impact of peer evaluation on patient outcomes. Best practices included standardized scoring systems, prospective criteria for case selection, and collaborative and non-judgmental review.ConclusionsPeer evaluation of invasive medical procedures is feasible and generally demonstrates good or excellent inter-observer agreement when performed with structured tools. Our review identifies common elements of successful interventions across specialties. However, there is limited evidence that peer-evaluated performance is linked to other quality measures or that feedback to clinicians improves patient care or outcomes. Additional research is needed to develop and test peer evaluation and feedback interventions.
| S-EPMC9335975 | biostudies-literature
Project description:Evaluation is an integral part of education in German medical schools. According to the quality standards set by the German Society for Evaluation, evaluation tools must provide an accurate and fair appraisal of teaching quality. Thus, data collection tools must be highly reliable and valid. This review summarises the current literature on evaluation of medical education with regard to the possible dimensions of teaching quality, the psychometric properties of survey instruments and potential confounding factors.We searched Pubmed, PsycINFO and PSYNDEX for literature on evaluation in medical education and included studies published up until June 30, 2011 as well as articles identified in the "grey literature". RESULTS are presented as a narrative review.We identified four dimensions of teaching quality: structure, process, teacher characteristics, and outcome. Student ratings are predominantly used to address the first three dimensions, and a number of reliable tools are available for this purpose. However, potential confounders of student ratings pose a threat to the validity of these instruments. Outcome is usually operationalised in terms of student performance on examinations, but methodological problems may limit the usability of these data for evaluation purposes. In addition, not all examinations at German medical schools meet current quality standards.The choice of tools for evaluating medical education should be guided by the dimension that is targeted by the evaluation. Likewise, evaluation results can only be interpreted within the context of the construct addressed by the data collection tool that was used as well as its specific confounding factors.
| S-EPMC4576315 | biostudies-literature
Project description:BACKGROUND:With the increasing recognition that leadership skills can be acquired, there is a heightened focus on incorporating leadership training as a part of graduate medical education. However, there is considerable lack of agreement regarding how to facilitate acquisition of these skills to resident, chief resident, and fellow physicians. METHODS:Articles were identified through a search of Ovid MEDLINE, EMBASE, CINAHL, ERIC, PsycNet, Cochrane Systemic Reviews, and Cochrane Central Register of Controlled Trials from 1948 to 2019. Additional sources were identified through contacting authors and scanning references. We included articles that described and evaluated leadership training programs in the United States and Canada. Methodological quality was assessed via the MERSQI (Medical Education Research Study Quality Instrument). RESULTS:Fifteen studies, which collectively included 639 residents, chief residents, and fellows, met the eligibility criteria. The format, content, and duration of these programs varied considerably. The majority focused on conflict management, interpersonal skills, and stress management. Twelve were prospective case series and three were retrospective. Seven used pre- and post-test surveys, while seven used course evaluations. Only three had follow-up evaluations after 6 months to 1 year. MERSQI scores ranged from 6 to 9. CONCLUSIONS:Despite interest in incorporating structured leadership training into graduate medical education curricula, there is a lack of methodologically rigorous studies evaluating its effectiveness. High-quality well-designed studies, focusing particularly on the validity of content, internal structure, and relationship to other variables, are required in order to determine if these programs have a lasting effect on the acquisition of leadership skills.
| S-EPMC7268469 | biostudies-literature
Project description:IntroductionFeedback from learners is known to be an important motivator for medical teachers, but it can be de-motivating if delivered poorly, leaving teachers frustrated and uncertain. Research has identified challenges learners face in providing upward feedback, but has not explored how challenges influence learners' goals and approaches to giving feedback. This study explored learner perspectives on providing feedback to teachers to advance understanding of how to optimize upward feedback quality.MethodsWe conducted semi-structured interviews with 16 learners from the MD program at the University of British Columbia. Applying an interpretive description methodology, interviews continued until data sufficiency was achieved. Iterative analysis accounted for general trends across seniority, site of training, age and gender as well as individual variations.FindingsLearners articulated well-intentioned goals in relation to upward feedback (e.g., to encourage effective teaching practices). However, conflicting priorities such as protecting one's image created tensions leading to feedback that was discordant with teaching quality. Several factors, including the number of feedback requests learners face and whether learners think their feedback is meaningful mediated the extent to which upward feedback goals or competing goals were enacted.DiscussionOur findings offer a nuanced understanding of the complexities that influence learners' approaches to upward feedback when challenges arise. In particular, goal conflicts make it difficult for learners to contribute to teacher support through upward feedback. Efforts to encourage the quality of upward feedback should begin with reducing competition between goals by addressing factors that mediate goal prioritization.
| S-EPMC10038106 | biostudies-literature
Project description:BackgroundValid cause of death data are essential for health policy formation. The quality of medical certification of cause of death (MCCOD) by physicians directly affects the utility of cause of death data for public policy and hospital management. Whilst training in correct certification has been provided for physicians and medical students, the impact of training is often unknown. This study was conducted to systematically review and meta-analyse the effectiveness of training interventions to improve the quality of MCCOD.MethodsThis review was registered in the International Prospective Register of Systematic Reviews (PROSPERO; Registration ID: CRD42020172547) and followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. CENTRAL, Ovid MEDLINE and Ovid EMBASE databases were searched using pre-defined search strategies covering the eligibility criteria. Studies were selected using four screening questions using the Distiller-SR software. Risk of bias assessments were conducted with GRADE recommendations and ROBINS-I criteria for randomised and non-randomised interventions, respectively. Study selection, data extraction and bias assessments were performed independently by two reviewers with a third reviewer to resolve conflicts. Clinical, methodological and statistical heterogeneity assessments were conducted. Meta-analyses were performed with Review Manager 5.4 software using the 'generic inverse variance method' with risk difference as the pooled estimate. A 'summary of findings' table was prepared using the 'GRADEproGDT' online tool. Sensitivity analyses and narrative synthesis of the findings were also performed.ResultsAfter de-duplication, 616 articles were identified and 21 subsequently selected for synthesis of findings; four underwent meta-analysis. The meta-analyses indicated that selected training interventions significantly reduced error rates among participants, with pooled risk differences of 15-33%. Robustness was identified with the sensitivity analyses. The findings of the narrative synthesis were similarly suggestive of favourable outcomes for both physicians and medical trainees.ConclusionsTraining physicians in correct certification improves the accuracy and policy utility of cause of death data. Investment in MCCOD training activities should be considered as a key component of strategies to improve vital registration systems given the potential of such training to substantially improve the quality of cause of death data.
| S-EPMC7728523 | biostudies-literature
Project description:BACKGROUND:A strong need exists for a validated tool that clearly defines peer review report quality in biomedical research, as it will allow evaluating interventions aimed at improving the peer review process in well-performed trials. We aim to identify and describe existing tools for assessing the quality of peer review reports in biomedical research. METHODS:We conducted a methodological systematic review by searching PubMed, EMBASE (via Ovid) and The Cochrane Methodology Register (via The Cochrane Library) as well as Google® for all reports in English describing a tool for assessing the quality of a peer review report in biomedical research. Data extraction was performed in duplicate using a standardized data extraction form. We extracted information on the structure, development and validation of each tool. We also identified quality components across tools using a systematic multi-step approach and we investigated quality domain similarities among tools by performing hierarchical, complete-linkage clustering analysis. RESULTS:We identified a total number of 24 tools: 23 scales and 1 checklist. Six tools consisted of a single item and 18 had several items ranging from 4 to 26. None of the tools reported a definition of 'quality'. Only 1 tool described the scale development and 10 provided measures of validity and reliability. Five tools were used as an outcome in a randomized controlled trial (RCT). Moreover, we classified the quality components of the 18 tools with more than one item into 9 main quality domains and 11 subdomains. The tools contained from two to seven quality domains. Some domains and subdomains were considered in most tools such as the detailed/thorough (11/18) nature of reviewer's comments. Others were rarely considered, such as whether or not the reviewer made comments on the statistical methods (1/18). CONCLUSION:Several tools are available to assess the quality of peer review reports; however, the development and validation process is questionable and the concepts evaluated by these tools vary widely. The results from this study and from further investigations will inform the development of a new tool for assessing the quality of peer review reports in biomedical research.
| S-EPMC6402095 | biostudies-literature
Project description:Patient feedback is considered integral to quality improvement and professional development. However, while popular across the educational continuum, evidence to support its efficacy in facilitating positive behaviour change in a postgraduate setting remains unclear. This review therefore aims to explore the evidence that supports, or refutes, the impact of patient feedback on the medical performance of qualified doctors.Electronic databases PubMed, EMBASE, Medline and PsycINFO were systematically searched for studies assessing the impact of patient feedback on medical performance published in the English language between 2006-2016. Impact was defined as a measured change in behaviour using Barr's (2000) adaptation of Kirkpatrick's four level evaluation model. Papers were quality appraised, thematically analysed and synthesised using a narrative approach.From 1,269 initial studies, 20 articles were included (qualitative (n=8); observational (n=6); systematic review (n=3); mixed methodology (n=1); randomised control trial (n=1); and longitudinal (n=1) design). One article identified change at an organisational level (Kirkpatrick level 4); six reported a measured change in behaviour (Kirkpatrick level 3b); 12 identified self-reported change or intention to change (Kirkpatrick level 3a), and one identified knowledge or skill acquisition (Kirkpatrick level 2). No study identified a change at the highest level, an improvement in the health and wellbeing of patients. The main factors found to influence the impact of patient feedback were: specificity; perceived credibility; congruence with physician self-perceptions and performance expectations; presence of facilitation and reflection; and inclusion of narrative comments. The quality of feedback facilitation and local professional cultures also appeared integral to positive behaviour change.Patient feedback can have an impact on medical performance. However, actionable change is influenced by several contextual factors and cannot simply be guaranteed. Patient feedback is likely to be more influential if it is specific, collected through credible methods and contains narrative information. Data obtained should be fed back in a way that facilitates reflective discussion and encourages the formulation of actionable behaviour change. A supportive cultural understanding of patient feedback and its intended purpose is also essential for its effective use.
| S-EPMC6069829 | biostudies-other
Project description:BACKGROUND:Empathy and compassion are vital components of health care quality; however, physicians frequently miss opportunities for empathy and compassion in patient care. Despite evidence that empathy and compassion training can be effective, the specific behaviors that should be taught remain unclear. We synthesized the biomedical literature on empathy and compassion training in medical education to find the specific curricula components (skills and behaviors) demonstrated to be effective. METHODS:We searched CENTRAL, MEDLINE, EMBASE, and CINAHL using a previously published comprehensive search strategy. We screened reference lists of the articles meeting inclusion criteria to identify additional studies for potential inclusion. Study inclusion criteria were: (1) intervention arm in which subjects underwent an educational curriculum aimed at enhancing empathy and/or compassion; (2) clearly defined control arm in which subjects did not receive the curriculum; (3) curriculum was tested on physicians (or physicians-in-training); and (4) outcome measure assessing the effect of the curriculum on physician empathy and/or compassion. We performed a qualitative analysis to collate and tabulate effects of tested curricula according to recommended methodology from the Cochrane Handbook. We used the Cochrane Collaboration's tool for assessing risk of bias. RESULTS:Fifty-two studies (total n = 5,316) met inclusion criteria. Most (75%) studies found that the tested curricula improved physician empathy and/or compassion on at least one outcome measure. We identified the following key behaviors to be effective: (1) sitting (versus standing) during the interview; (2) detecting patients' non-verbal cues of emotion; (3) recognizing and responding to opportunities for compassion; (4) non-verbal communication of caring (e.g. eye contact); and (5) verbal statements of acknowledgement, validation, and support. These behaviors were found to improve patient perception of physician empathy and/or compassion. CONCLUSION:Evidence suggests that training can enhance physician empathy and compassion. Training curricula should incorporate the specific behaviors identified in this report.
| S-EPMC6705835 | biostudies-literature