Dataset Information

Impact of workplace based assessment on doctors' education and performance: a systematic review.

ABSTRACT:

Objective

To investigate the literature for evidence that workplace based assessment affects doctors' education and performance.

Design

Systematic review.

Data sources

The primary data sources were the databases Journals@Ovid, Medline, Embase, CINAHL, PsycINFO, and ERIC. Evidence based reviews (Bandolier, Cochrane Library, DARE, HTA Database, and NHS EED) were accessed and searched via the Health Information Resources website. Reference lists of relevant studies and bibliographies of review articles were also searched. Review methods Studies of any design that attempted to evaluate either the educational impact of workplace based assessment, or the effect of workplace based assessment on doctors' performance, were included. Studies were excluded if the sampled population was non-medical or the study was performed with medical students. Review articles, commentaries, and letters were also excluded. The final exclusion criterion was the use of simulated patients or models rather than real life clinical encounters.

Results

Sixteen studies were included. Fifteen of these were non-comparative descriptive or observational studies; the other was a randomised controlled trial. Study quality was mixed. Eight studies examined multisource feedback with mixed results; most doctors felt that multisource feedback had educational value, although the evidence for practice change was conflicting. Some junior doctors and surgeons displayed little willingness to change in response to multisource feedback, whereas family physicians might be more prepared to initiate change. Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses. Four studies examined the mini-clinical evaluation exercise, one looked at direct observation of procedural skills, and three were concerned with multiple assessment methods: all these studies reported positive results for the educational impact of workplace based assessment tools. However, there was no objective evidence of improved performance with these tools.

Conclusions

Considering the emphasis placed on workplace based assessment as a method of formative performance assessment, there are few published articles exploring its impact on doctors' education and performance. This review shows that multisource feedback can lead to performance improvement, although individual factors, the context of the feedback, and the presence of facilitation have a profound effect on the response. There is no evidence that alternative workplace based assessment tools (mini-clinical evaluation exercise, direct observation of procedural skills, and case based discussion) lead to improvement in performance, although subjective reports on their educational impact are positive.

SUBMITTER: Miller A

PROVIDER: S-EPMC2945627 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:OBJECTIVES:The well-being of doctors is at risk, as evidenced by high burnout rates amongst doctors around the world. Alarmingly, burned-out doctors are more likely to exhibit low levels of professionalism and provide suboptimal patient care. Research suggests that burnout and the well-being of doctors can be improved by mindfulness-based interventions (MBIs). Furthermore, MBIs may improve doctors' performance (eg in empathy). However, there are no published systematic reviews that clarify the effects of MBIs on doctor well-being or performance to inform future research and professional development programmes. We therefore systematically reviewed and narratively synthesised findings on the impacts of MBIs on doctors' well-being and performance. METHODS:We searched PubMed and PsycINFO from inception to 9 May 2018 and independently reviewed studies investigating the effects of MBIs on doctor well-being or performance. We systematically extracted data and assessed study quality according to the Medical Education Research Study Quality Instrument (MERSQI), and narratively reported study findings. RESULTS:We retrieved a total of 934 articles, of which 24 studies met our criteria; these included randomised, (un)controlled or qualitative studies of average quality. Effects varied across MBIs with different training contents or formats: MBIs including essential mindfulness training elements, or employing group-based training, mostly showed positive effects on the well-being or performance of doctors across different educational and hospital settings. Doctors perceived both benefits (enhanced self- and other-understanding) and challenges (time limitations and feasibility) associated with MBIs. Findings were subject to the methodological limitations of studies (eg the use of self-selected participants, lack of placebo interventions, use of self-reported outcomes). CONCLUSIONS:This review indicates that doctors can perceive positive impacts of MBIs on their well-being and performance. However, the evidence was subject to methodological limitations and does not yet support the standardisation of MBIs in professional development programmes. Rather, health care organisations could consider including group-based MBIs as voluntary modules for doctors with specific well-being needs or ambitions regarding professional development.

Project description:Patient feedback is considered integral to quality improvement and professional development. However, while popular across the educational continuum, evidence to support its efficacy in facilitating positive behaviour change in a postgraduate setting remains unclear. This review therefore aims to explore the evidence that supports, or refutes, the impact of patient feedback on the medical performance of qualified doctors.Electronic databases PubMed, EMBASE, Medline and PsycINFO were systematically searched for studies assessing the impact of patient feedback on medical performance published in the English language between 2006-2016. Impact was defined as a measured change in behaviour using Barr's (2000) adaptation of Kirkpatrick's four level evaluation model. Papers were quality appraised, thematically analysed and synthesised using a narrative approach.From 1,269 initial studies, 20 articles were included (qualitative (n=8); observational (n=6); systematic review (n=3); mixed methodology (n=1); randomised control trial (n=1); and longitudinal (n=1) design). One article identified change at an organisational level (Kirkpatrick level 4); six reported a measured change in behaviour (Kirkpatrick level 3b); 12 identified self-reported change or intention to change (Kirkpatrick level 3a), and one identified knowledge or skill acquisition (Kirkpatrick level 2). No study identified a change at the highest level, an improvement in the health and wellbeing of patients. The main factors found to influence the impact of patient feedback were: specificity; perceived credibility; congruence with physician self-perceptions and performance expectations; presence of facilitation and reflection; and inclusion of narrative comments. The quality of feedback facilitation and local professional cultures also appeared integral to positive behaviour change.Patient feedback can have an impact on medical performance. However, actionable change is influenced by several contextual factors and cannot simply be guaranteed. Patient feedback is likely to be more influential if it is specific, collected through credible methods and contains narrative information. Data obtained should be fed back in a way that facilitates reflective discussion and encourages the formulation of actionable behaviour change. A supportive cultural understanding of patient feedback and its intended purpose is also essential for its effective use.

Project description:ObjectivesTo summarise evidence on intrawork breaks and their associated effect on doctors' well-being and/or performance at work.DesignSystematic review following Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 statement guidelines DATA SOURCES: Embase, PubMed, Web of Science (Core Collection) and PsychINFO were systematically searched on 6 June 2021.Eligibility criteriaNo restrictions were placed on language, study design or date of publication.Data extraction and analysisMethodological quality was appraised using Cochrane's Risk of Bias (ROB-2), Cochrane's Risk of Bias in Non-randomised Studies (ROBINS-I), and the Johanna Briggs Institute (JBI) checklists for cross-sectional, cohort and qualitative studies. Quantitative synthesis was not undertaken due to substantial heterogeneity of design and outcomes. Results are presented narratively.ResultsDatabase searches returned 10 557 results and searches of other sources returned two additional records. Thirty-two papers were included in the systematic review, comprised of 29 unique studies, participants and topics and 3 follow-up studies. A variety of well-being and performance outcome measures were used. Overall, findings indicate that intrawork breaks improved some measures of well-being and/or work performance. However, methodological quality was judged to be low with a high risk of bias in most included studies.DiscussionUsing existing evidence, it is not possible to conclude with confidence whether intrawork breaks improve well-being and/or work performance in doctors. There is much inconsistency regarding how breaks are defined, measured and the outcomes used to assess effectiveness. Future research should seek to: (a) define and standardise the measurement of breaks, (b) use valid, reliable outcome measures to evaluate their impact on well-being and performance and (c) minimise the risk of bias in studies where possible.Prospero registration numberCRD42020156924; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=156924.

Project description:BACKGROUND:The widening gap between innovations in the medical field and the dissemination of such information to doctors may affect the quality of care. Offline computer-based digital education (OCDE) may be a potential solution to overcoming the geographical, financial, and temporal obstacles faced by doctors. OBJECTIVE:The objectives of this systematic review were to evaluate the effectiveness of OCDE compared with face-to-face learning, no intervention, or other types of digital learning for improving medical doctors' knowledge, cognitive skills, and patient-related outcomes. Secondary objectives were to assess the cost-effectiveness (CE) of OCDE and any adverse effects. METHODS:We searched major bibliographic databases from 1990 to August 2017 to identify relevant articles and followed the Cochrane methodology for systematic reviews of intervention. RESULTS:Overall, 27 randomized controlled trials (RCTs), 1 cluster RCT (cRCT), and 1 quasi-RCT were included in this review. The total number of participants was 1690 in addition to the cRCT, which included 24 practices. Due to the heterogeneity of the participants, interventions, and outcomes, meta-analysis was not feasible, and the results were presented as narrative summary. Compared with face-to-face learning, the effect of OCDE on knowledge gain is uncertain (ratio of the means [RM] range 0.95-1.17; 8 studies, 495 participants; very low grade of evidence). From the same comparison, the effect of OCDE on cognitive skill gain is uncertain (RM range 0.1-0.9; 8 studies, 375 participants; very low grade of evidence). OCDE may have little or no effect on patients' outcome compared with face-to-face education (2 studies, 62 participants; low grade of evidence). Compared with no intervention, OCDE may improve knowledge gain (RM range 1.36-0.98; 4 studies, 401 participants; low grade of evidence). From the same comparison, the effect of OCDE on cognitive skill gain is uncertain (RM range 1.1-1.15; 4 trials, 495 participants; very low grade of evidence). One cRCT, involving 24 practices, investigated patients' outcome in this comparison and showed no difference between the 2 groups with low-grade evidence. Compared with text-based learning, the effect of OCDE on cognitive skills gain is uncertain (RM range 0.91-1.46; 3 trials with 4 interventions; 68 participants; very low-grade evidence). No study in this comparison investigated knowledge gain or patients' outcomes. One study assessed the CE and showed that OCDE was cost-effective when compared with face-to-face learning if the cost is less than or equal to Can $200. No trial evaluated the adverse effect of OCDE. CONCLUSIONS:The effect of OCDE compared with other methods of education on medical doctors' knowledge and cognitive skill gain is uncertain. OCDE may improve doctors' knowledge compared with no intervention but its effect on doctors' cognitive skills is uncertain. OCDE may have little or no effect in improving patients' outcome.

Project description:BACKGROUND:Globally, online and local area network-based (LAN) digital education (ODE) has grown in popularity. Blended learning is used by ODE along with traditional learning. Studies have shown the increasing potential of these technologies in training medical doctors; however, the evidence for its effectiveness and cost-effectiveness is unclear. OBJECTIVE:This systematic review evaluated the effectiveness of online and LAN-based ODE in improving practicing medical doctors' knowledge, skills, attitude, satisfaction (primary outcomes), practice or behavior change, patient outcomes, and cost-effectiveness (secondary outcomes). METHODS:We searched seven electronic databased for randomized controlled trials, cluster-randomized trials, and quasi-randomized trials from January 1990 to March 2017. Two review authors independently extracted data and assessed the risk of bias. We have presented the findings narratively. We mainly compared ODE with self-directed/face-to-face learning and blended learning with self-directed/face-to-face learning. RESULTS:A total of 93 studies (N=16,895) were included, of which 76 compared ODE (including blended) and self-directed/face-to-face learning. Overall, the effect of ODE (including blended) on postintervention knowledge, skills, attitude, satisfaction, practice or behavior change, and patient outcomes was inconsistent and ranged mostly from no difference between the groups to higher postintervention score in the intervention group (small to large effect size, very low to low quality evidence). Twenty-one studies reported higher knowledge scores (small to large effect size and very low quality) for the intervention, while 20 studies reported no difference in knowledge between the groups. Seven studies reported higher skill score in the intervention (large effect size and low quality), while 13 studies reported no difference in the skill scores between the groups. One study reported a higher attitude score for the intervention (very low quality), while four studies reported no difference in the attitude score between the groups. Four studies reported higher postintervention physician satisfaction with the intervention (large effect size and low quality), while six studies reported no difference in satisfaction between the groups. Eight studies reported higher postintervention practice or behavior change for the ODE group (small to moderate effect size and low quality), while five studies reported no difference in practice or behavior change between the groups. One study reported higher improvement in patient outcome, while three others reported no difference in patient outcome between the groups. None of the included studies reported any unintended/adverse effects or cost-effectiveness of the interventions. CONCLUSIONS:Empiric evidence showed that ODE and blended learning may be equivalent to self-directed/face-to-face learning for training practicing physicians. Few other studies demonstrated that ODE and blended learning may significantly improve learning outcomes compared to self-directed/face-to-face learning. The quality of the evidence in these studies was found to be very low for knowledge. Further high-quality randomized controlled trials are required to confirm these findings.

Project description:OBJECTIVE:Synthesise evidence about the impact of family medicine/general practice (FM) clerkships on undergraduate medical students, teaching general/family practitioners (FPs) and/or their patients. DATA SOURCES:Medline, ERIC, PsycINFO, EMBASE and Web of Knowledge searched from 21 November to 17 December 2013. Primary, empirical, quantitative or qualitative studies, since 1990, with abstracts included. No country restrictions. Full text languages: English, French, Spanish, German, Dutch or Italian. REVIEW METHODS:Independent selection and data extraction by two authors using predefined data extraction fields, including Kirkpatrick's levels for educational intervention outcomes, study quality indicators and Best Evidence Medical Education (BEME) strength of findings' grades. Descriptive narrative synthesis applied. RESULTS:Sixty-four included articles: impact on students (48), teaching FPs (12) and patients (8). Sample sizes: 16-1095 students, 3-146 FPs and 94-2550 patients. Twenty-six studies evaluated at Kirkpatrick level 1, 26 at level 2 and 6 at level 3. Only one study achieved BEME's grade 5. The majority was assessed as grade 4 (27) and 3 (33). Students reported satisfaction with content and process of teaching as well as learning in FM clerkships. They enhanced previous learning, and provided unique learning on dealing with common acute and chronic conditions, health maintenance, disease prevention, communication and problem-solving skills. Students' attitudes towards FM were improved, but new or enhanced interest in FM careers did not persist without change after graduation. Teaching FPs reported increased job satisfaction and stimulation for professional development, but also increased workload and less productivity, depending on the setting. Overall, student's presence and participation did not have a negative impact on patients. CONCLUSIONS:Research quality on the impact of FM clerkships is still limited, yet across different settings and countries, positive impact is reported on students, FPs and patients. Future studies should involve different stakeholders, medical schools and countries, and use standardised and validated evaluation tools.

Project description:Evaluating the patient impact of health professions education is a societal priority with many challenges. Researchers would benefit from a summary of topics studied and potential methodological problems. We sought to summarize key information on patient outcomes identified in a comprehensive systematic review of simulation-based instruction.Systematic search of MEDLINE, EMBASE, CINAHL, PsychINFO, Scopus, key journals, and bibliographies of previous reviews through May 2011.Original research in any language measuring the direct effects on patients of simulation-based instruction for health professionals, in comparison with no intervention or other instruction.Two reviewers independently abstracted information on learners, topics, study quality including unit of analysis, and validity evidence. We pooled outcomes using random effects.From 10,903 articles screened, we identified 50 studies reporting patient outcomes for at least 3,221 trainees and 16,742 patients. Clinical topics included airway management (14 studies), gastrointestinal endoscopy (12), and central venous catheter insertion (8). There were 31 studies involving postgraduate physicians and seven studies each involving practicing physicians, nurses, and emergency medicine technicians. Fourteen studies (28 %) used an appropriate unit of analysis. Measurement validity was supported in seven studies reporting content evidence, three reporting internal structure, and three reporting relations with other variables. The pooled Hedges' g effect size for 33 comparisons with no intervention was 0.47 (95 % confidence interval [CI], 0.31-0.63); and for nine comparisons with non-simulation instruction, it was 0.36 (95 % CI, -0.06 to 0.78).Focused field in education; high inconsistency (I(2)?>?50 % in most analyses).Simulation-based education was associated with small-moderate patient benefits in comparison with no intervention and non-simulation instruction, although the latter did not reach statistical significance. Unit of analysis errors were common, and validity evidence was infrequently reported.

Project description:BackgroundResidents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance.ObjectiveThis study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings.MethodsNLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported.ResultsThe database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%).ConclusionsThe ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.