Dataset Information

Repeated analyses of national clinical audit reports demonstrate improvements in feedback methods.

ABSTRACT: Background: There is growing interest in the impact of national clinical audit programmes on the quality of healthcare. There is also an evolving evidence-base for enhancing the design and delivery of audit and feedback. We assessed the extent to which a sample of UK national clinical audit feedback reports met a set of good practice criteria over three time points.

Methods: We undertook three cross-sectional content analyses. We developed good practice criteria for the content and delivery of feedback based upon evidence, behavioural theory and expert opinion. We applied these to a feedback reports from 23 national audits listed on the Healthcare Quality Improvement Partnership (HQIP) website in November 2015. We repeated our assessments in January 2017 for 20 repeat feedback reports, after HQIP had published reporting guidance for national audits, and in August 2019 for a further 14 repeat feedback reports. We verified our assessments, where possible, with audit leads.

Results: Feedback reports consistently included strengths at baseline, including past or planned repeated audit cycles (21; 91%), stating the importance of the topic in relation to patient care (22; 93%), using multi-modal data presentation (23; 100%), and summarising key findings (23; 100%). We observed improvements over subsequent assessments, so that by 2019, at least 13 out of 14 (93%) feedback reports presented easily identifiable key findings and recommendations, linked recommendations to audit standards, and proposed easily identifiable action plans. Whilst the use of regional comparators did not improve, audit leads highlighted that programmes now provide local data via additional means. The main shortcoming was the time lag between data collection and feedback; none of the 14 reports assessed in 2019 presented performance data less than 6 months old. Audit leads highlighted that some of these data might be available via programme websites.

Conclusion: We identified increased adherence to good practice in feedback by national clinical audit programmes that may enhance their impact on service delivery and outcomes. There is scope for improvement, especially in the recency of performance data. With further refinements, a criterion-based assessment offers an efficient means of monitoring the quality of national clinical audit feedback reports.

SUBMITTER: Khan T

PROVIDER: S-EPMC7691059 | biostudies-literature | 2020 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Repeated analyses of national clinical audit reports demonstrate improvements in feedback methods.

Khan Tasneem T Alderson Sarah S Francis Jill J JJ Lorencatto Fabiana F Grant-Casey John J Stanworth Simon J SJ Foy Robbie R

Implementation science communications 20201125 1

<h4>Background</h4>There is growing interest in the impact of national clinical audit programmes on the quality of healthcare. There is also an evolving evidence-base for enhancing the design and delivery of audit and feedback. We assessed the extent to which a sample of UK national clinical audit feedback reports met a set of good practice criteria over three time points.<h4>Methods</h4>We undertook three cross-sectional content analyses. We developed good practice criteria for the content and ...[more]

PMID: 33292847

Similar Datasets

Project description:BackgroundAudit and feedback (A&F) is a common quality improvement strategy with highly variable effects on patient care. It is unclear how A&F effectiveness can be maximised. Since the core mechanism of action of A&F depends on drawing attention to a discrepancy between actual and desired performance, we aimed to understand current and best practices in the choice of performance comparator.MethodsWe described current choices for performance comparators by conducting a secondary review of randomised trials of A&F interventions and identifying the associated mechanisms that might have implications for effective A&F by reviewing theories and empirical studies from a recent qualitative evidence synthesis.ResultsWe found across 146 trials that feedback recipients' performance was most frequently compared against the performance of others (benchmarks; 60.3%). Other comparators included recipients' own performance over time (trends; 9.6%) and target standards (explicit targets; 11.0%), and 13% of trials used a combination of these options. In studies featuring benchmarks, 42% compared against mean performance. Eight (5.5%) trials provided a rationale for using a specific comparator. We distilled mechanisms of each comparator from 12 behavioural theories, 5 randomised trials, and 42 qualitative A&F studies.ConclusionClinical performance comparators in published literature were poorly informed by theory and did not explicitly account for mechanisms reported in qualitative studies. Based on our review, we argue that there is considerable opportunity to improve the design of performance comparators by (1) providing tailored comparisons rather than benchmarking everyone against the mean, (2) limiting the amount of comparators being displayed while providing more comparative information upon request to balance the feedback's credibility and actionability, (3) providing performance trends but not trends alone, and (4) encouraging feedback recipients to set personal, explicit targets guided by relevant information.

Project description:Clinical narratives are a valuable source of information for both patient care and biomedical research. Given the unstructured nature of medical reports, specific automatic techniques are required to extract relevant entities from such texts. In the natural language processing (NLP) community, this task is often addressed by using supervised methods. To develop such methods, both reliably-annotated corpora and elaborately designed features are needed. Despite the recent advances on corpora collection and annotation, research on multiple domains and languages is still limited. In addition, to compute the features required for supervised classification, suitable language- and domain-specific tools are needed. In this work, we propose a novel application of recurrent neural networks (RNNs) for event extraction from medical reports written in Italian. To train and evaluate the proposed approach, we annotated a corpus of 75 cardiology reports for a total of 4365 mentions of relevant events and their attributes (e.g., the polarity). For the annotation task, we developed specific annotation guidelines, which are provided together with this paper. The RNN-based classifier was trained on a training set including 3335 events (60 documents). The resulting model was integrated into an NLP pipeline that uses a dictionary lookup approach to search for relevant concepts inside the text. A test set of 1030 events (15 documents) was used to evaluate and compare different pipeline configurations. As a main result, using the RNN-based classifier instead of the dictionary lookup approach allowed increasing recall from 52.4% to 88.9%, and precision from 81.1% to 88.2%. Further, using the two methods in combination, we obtained final recall, precision, and F1 score of 91.7%, 88.6%, and 90.1%, respectively. These experiments indicate that integrating a well-performing RNN-based classifier with a standard knowledge-based approach can be a good strategy to extract information from clinical text in non-English languages.

Project description:BackgroundIn Ontario, Canada, a government agency known as Ontario Health is responsible for making audit and feedback reports available to all family physicians to encourage ongoing quality improvement. The confidential report provides summary data on 3 key areas of practice: safe prescription, cancer screening, and diabetes management.ObjectiveThis report was redesigned to improve its usability in line with evidence. The objective of this study was to explore how the redesign was perceived, with an emphasis on recipients' understanding of the report and their engagement with it.MethodsWe conducted qualitative semistructured interviews with family physicians who had experience with both versions of the report recruited through purposeful and snowball sampling. We analyzed the transcripts following an emergent and iterative approach.ResultsSaturation was reached after 17 family physicians participated. In total, 2 key themes emerged as factors that affected the perceived usability of the report: alignment between the report and the recipients' expectations and capacity to engage in quality improvement. Family physicians expected the report and its quality indicators to reflect best practices and to be valid and accurate. They also expected the report to offer feedback on the clinical activities they perceived to be within their control to change. Furthermore, family physicians expected the goal of the report to be aligned with their perspective on feasible quality improvement activities. Most of these expectations were not met, limiting the perceived usability of the report. The capacity to engage with audit and feedback was hindered by several organizational and physician-level barriers, including the lack of fit with the existing workflow, competing priorities, time constraints, and insufficient skills for bridging the gaps between their data and the corresponding desired actions.ConclusionsDespite recognized improvements in the design of the report to better align with best practices, it was not perceived as highly usable. Improvements in the presentation of the data could not overcome misalignment with family physicians' expectations or the limited capacity to engage with the report. Integrating iterative evaluations informed by user-centered design can complement evidence-based guidance for implementation strategies. Creating a space for bringing together audit and feedback designers and recipients may help improve usability and effectiveness.

Project description:BackgroundThe Department of Veterans Affairs (VA) has led the industry in measuring facility performance as a critical element in improving quality of care, investing substantial resources to develop and maintain valid and cost-effective measures. The External Peer Review Program (EPRP) of the VA is the official data source for monitoring facility performance, used to prioritize the quality areas needing most attention. Facility performance measurement has significantly improved preventive and chronic care, as well as overall quality; however, much variability still exists in levels of performance across measures and facilities. Audit and feedback (A&F), an important component of effective performance measurement, can help reduce this variability and improve overall performance. Previous research suggests that VA Medical Centers (VAMCs) with high EPRP performance scores tend to use EPRP data as a feedback source. However, the manner in which EPRP data are used as a feedback source by individual providers as well as service line, facility, and network leadership is not well understood. An in-depth understanding of mental models, strategies, and specific feedback process characteristics adopted by high-performing facilities is thus urgently needed.This research compares how leaders of high, low, and moderately performing VAMCs use clinical performance data from the EPRP as a feedback tool to maintain and improve quality of care.MethodsWe will conduct a qualitative, grounded theory analysis of up to 64 interviews using a novel method of sampling primary care, facility, and Veterans Integrated Service Network (VISN) leadership at high-, moderate-, and low-performing facilities. We will analyze interviews for evidence of cross-facility differences in perceptions of performance data usefulness and strategies for disseminating performance data evaluating performance, with particular attention to timeliness, individualization, and punitiveness of feedback delivery.DiscussionMost research examining feedback to improve provider and facility performance lacks a detailed understanding of the elements of effective feedback. This research will highlight the elements most commonly used at high-performing facilities and identify additional features of their successful feedback strategies not previously identified. Armed with this information, practices can implement more effective A&F interventions to improve quality of care.

Project description:BackgroundThere is considerable evidence about the effectiveness of audit coupled with feedback, although few audit with feedback interventions have been conducted in long-term care (LTC) settings to date. In general, the effects have been found to be modest at best, although in settings where there has been little history of audit and feedback, the effects may be greater, at least initially. The primary purpose of the Data for Improvement and Clinical Excellence (DICE) Long-Term Care project is to assess the effects of an audit with feedback intervention delivered monthly over 13 months in four LTC facilities. The research questions we addressed are:1. What effects do feedback reports have on processes and outcomes over time?2. How do different provider groups in LTC and home care respond to feedback reports based on data targeted at improving quality of care?Methods/designThe research team conducting this study comprises researchers and decision makers in continuing care in the province of Alberta, Canada. The intervention consists of monthly feedback reports in nine LTC units in four facilities in Edmonton, Alberta. Data for the feedback reports comes from the Resident Assessment Instrument Minimum Data Set (RAI) version 2.0, a standardized instrument mandated for use in LTC facilities throughout Alberta. Feedback reports consist of one page, front and back, presenting both graphic and textual information. Reports are delivered to all staff working in the four LTC facilities. The primary evaluation uses a controlled interrupted time series design both adjusted and unadjusted for covariates. The concurrent process evaluation uses observation and self-report to assess uptake of the feedback reports. Following the project phase described in this protocol, a similar intervention will be conducted in home care settings in Alberta. Depending on project findings, if they are judged useful by decision makers participating in this research team, we plan dissemination and spread of the feedback report approach throughout Alberta.

Project description:BackgroundAudit and Feedback (A&F) is a widely used quality improvement technique that measures clinicians' clinical performance and reports it back to them. Computerised A&F (e-A&F) system interfaces may consist of four key components: (1) Summaries of clinical performance; (2) Patient lists; (3) Patient-level data; (4) Recommended actions. There is a lack of evidence regarding how to best design e-A&F interfaces; establishing such evidence is key to maximising usability, and in turn improving patient safety.AimTo evaluate the usability of a novel theoretically-informed and research-led e-A&F system for primary care (the Performance Improvement plaN GeneratoR: PINGR).Objectives(1) Describe PINGR's design, rationale and theoretical basis; (2) Identify usability issues with PINGR; (3) Understand how these issues may interfere with the cognitive goals of end-users; (4) Translate the issues into recommendations for the user-centred design of e-A&F systems.MethodsEight experienced health system evaluators performed a usability inspection using an innovative hybrid approach consisting of five stages: (1) Development of representative user tasks, Goals, and Actions; (2) Combining Heuristic Evaluation and Cognitive Walkthrough methods into a single protocol to identify usability issues; (3) Consolidation of issues; (4) Severity rating of consolidated issues; (5) Analysis of issues according to usability heuristics, interface components, and Goal-Action structure.ResultsA final list of 47 issues were categorised into 8 heuristic themes. The most error-prone heuristics were 'Consistency and standards' (13 usability issues; 28% of the total) and 'Match between system and real world' (n=10, 21%). The recommended actions component of the PINGR interface had the most usability issues (n=21, 45%), followed by patient-level data (n=5, 11%), patient lists (n=4, 9%), and summaries of clinical performance (n=4, 9%). The most error-prone Actions across all user Goals were: (1) Patient selection from a list; (2) Data identification from a figure (both population-level and patient-level); (3) Disagreement with a system recommendation.ConclusionsBy contextualising our findings within the wider literature on health information system usability, we provide recommendations for the design of e-A&F system interfaces relating to their four key components, in addition to how they may be integrated within a system.

Dataset Information

Repeated analyses of national clinical audit reports demonstrate improvements in feedback methods.

Publications

Repeated analyses of national clinical audit reports demonstrate improvements in feedback methods.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets