Dataset Information

Developing Criteria and Associated Instructions for Consistent and Useful Quality Improvement Study Data Extraction for Health Systems.

ABSTRACT:

Background

The Agency for Healthcare Research and Quality (AHRQ) could devote resources to collate and assess quality improvement studies to support learning health systems (LHS) but there is no reliable data on the consistency of data extraction for important criteria.

Methods

We identified quality improvement studies and evaluated the consistency of data extraction from two experienced independent reviewers at three time points: baseline, first revision (where explicit instructions for each criterion were created), and final revision (where the instructions were revised). Six investigators looked at the data extracted by the two systematic reviewers and determined the extent of similarity on a scale of 0 to 10 (where 0 represented no similarity and 10 perfect similarity). There were 42 assessments for baseline, 42 assessments for the first revision, and 42 assessments for the final revision. We asked two LHS participants to assess the relative value of our criteria.

Results

The consistency of extraction improved from 1.17 ± 1.85 at baseline to 6.07 ± 2.76 after revision 1 (P < 0.001) and to 6.81 ± 1.94 out of 10 for the final revision (P < 0.001). However, the final revision was not significantly improved over the first revision (P = 0.14). One key informant rated the difficulty in finding and using quality improvement studies a 6 (moderately difficult) while the other a 4 (moderately difficult). When asked how valuable it would be if AHRQ found and collated the demographic information about the health systems and the interventions used in published quality improvement studies, they rated it a 9 (highly valuable) and a 6 (moderately valuable).

Conclusion

Creating explicit instructions for extracting data for quality improvement studies helps enhance the consistency of data extraction. This is important because it is difficult for LHS to vet these quality improvement studies on their own and they would value AHRQ's support in that regard.

SUBMITTER: Hernandez AV

PROVIDER: S-EPMC7652974 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectiveValid, reliable critical appraisal tools advance quality improvement (QI) intervention impacts by helping stakeholders identify higher quality studies. QI approaches are diverse and differ from clinical interventions. Widely used critical appraisal instruments do not take unique QI features into account and existing QI tools (eg, Standards for QI Reporting Excellence) are intended for publication guidance rather than critical appraisal. This study developed and psychometrically tested a critical appraisal instrument, the QI Minimum Quality Criteria Set (QI-MQCS) for assessing QI-specific features of QI publications.MethodsApproaches to developing the tool and ensuring validity included a literature review, in-person and online survey expert panel input, and application to empirical examples. We investigated psychometric properties in a set of diverse QI publications (N=54) by analysing reliability measures and item endorsement rates and explored sources of disagreement between reviewers.ResultsThe QI-MQCS includes 16 content domains to evaluate QI intervention publications: Organisational Motivation, Intervention Rationale, Intervention Description, Organisational Characteristics, Implementation, Study Design, Comparator Description, Data Sources, Timing, Adherence/Fidelity, Health Outcomes, Organisational Readiness, Penetration/Reach, Sustainability, Spread and Limitations. Median inter-rater agreement for QI-MQCS items was κ 0.57 (83% agreement). Item statistics indicated sufficient ability to differentiate between publications (median quality criteria met 67%). Internal consistency measures indicated coherence without excessive conceptual overlap (absolute mean interitem correlation=0.19). The critical appraisal instrument is accompanied by a user manual detailing What to consider, Where to look and How to rate.ConclusionsWe developed a ready-to-use, valid and reliable critical appraisal instrument applicable to healthcare QI intervention publications, but recognise scope for continuing refinement.

Project description:ObjectiveDespite having a detailed MRI-safety questionnaire check at the point of referral, we have encountered a significant number of near-misses with patients being identified with MRI-Unsafe devices at the time of appointments, making this an important safety hazard.Methods and materialsA two-part survey was performed to assess referrer compliance of asking MRI-questionnaires. 120 outpatients across 3 MRI sites were interviewed at the time of appointment to confirm whether their referrers completed the MRI questionnaires with them at the time of referral.Location:Department of Radiology, Ninewells Hospital, Perth Royal Infirmary and Stracathro Hospital in Scotland.ResultsOnly 50-55 % of patients confirmed that they were asked about presence of a pacemaker at the point of referral. Less than 50 % of patients reported being asked about other potential hazards.Suggested strategies for change: (1) Risk Alert-Sent to all MRI referrers in the organization. ( 2) Changes to MRI Safety Questionnaire. (3) Feedback mechanism to referrers-NHS trust website publications on number of recorded near-misses and wasted appointments due to MRI-safety issues. (4) Compulsory education/training of future referrers (junior doctors/allied health professionals). (5) Education of patients/public on MRI safety-Displaying patient information leaflets/posters in waiting areas of the hospital.Key measures for improvement: (1) Reduction in number of recorded near-misses. (2) System improvements, referrer and patient education, reduction of wasted MRI appointments and improvement of waiting-times for MRI appointments Effects of survey and conclusions: The survey highlights the possibility of inadequate referrer attention, and poor patient communication about MRI safety questionnaire with regards to potential hazards of MRI examination in presence of undeclared implants. It initiated several interventions resulting in improved patient safety, with no events in next 12 months, whilst promoting public and referrer's understanding of potential MRI safety issues. Such actions are recommended for all NHS centres across UK since there are significant similarities in functioning across UK.

Project description:BackgroundRecent policy reforms encourage quality improvement (QI) innovations in primary care, but practitioners lack clear guidance regarding spread inside organizations.PurposeWe designed this study to identify how large organizations can facilitate intraorganizational spread of QI innovations.Methodology/approachWe conducted ethnographic observation and interviews in a large, multispecialty, community-based medical group that implemented three QI innovations across 10 primary care sites using a new method for intraorganizational process development and spread. We compared quantitative outcomes achieved through the group's traditional versus new method, created a process model describing the steps in the new method, and identified barriers and facilitators at each step.FindingsThe medical group achieved substantial improvement using its new method of intraorganizational process development and spread of QI innovations: standard work for rooming and depression screening, vaccine error rates and order compliance, and Pap smear error rates. Our model details nine critical steps for successful intraorganizational process development (set priorities, assess the current state, develop the new process, and measure and refine) and spread (develop support, disseminate information, facilitate peer-to-peer training, reinforce, and learn and adapt). Our results highlight the importance of utilizing preexisting organizational structures such as established communication channels, standardized roles, common workflows, formal authority, and performance measurement and feedback systems when developing and spreading QI processes inside an organization. In particular, we detail how formal process advocate positions in each site for each role can facilitate the spread of new processes.Practice implicationsSuccessful intraorganizational spread is possible and sustainable. Developing and spreading new QI processes across sites inside an organization requires creating a shared understanding of the necessary process steps, considering the barriers that may arise at each step, and leveraging preexisting organizational structures to facilitate intraorganizational process development and spread.

Project description:INTRODUCTION:'Systems thinking' is often recommended in healthcare to support quality and safety activities but a shared understanding of this concept and purposeful guidance on its application are limited. Healthcare systems have been described as complex where human adaptation to localised circumstances is often necessary to achieve success. Principles for managing and improving system safety developed by the European Organisation for the Safety of Air Navigation (EUROCONTROL; a European intergovernmental air navigation organisation) incorporate a 'Safety-II systems approach' to promote understanding of how safety may be achieved in complex work systems. We aimed to adapt and contextualise the core principles of this systems approach and demonstrate the application in a healthcare setting. METHODS:The original EUROCONTROL principles were adapted using consensus-building methods with front-line staff and national safety leaders. RESULTS:Six interrelated principles for healthcare were agreed. The foundation concept acknowledges that 'most healthcare problems and solutions belong to the system'. Principle 1 outlines the need to seek multiple perspectives to understand system safety. Principle 2 prompts us to consider the influence of prevailing work conditions-demand, capacity, resources and constraints. Principle 3 stresses the importance of analysing interactions and work flow within the system. Principle 4 encourages us to attempt to understand why professional decisions made sense at the time and principle 5 prompts us to explore everyday work including the adjustments made to achieve success in changing system conditions.A case study is used to demonstrate the application in an analysis of a system and in the subsequent improvement intervention design. CONCLUSIONS:Application of the adapted principles underpins, and is characteristic of, a holistic systems approach and may aid care team and organisational system understanding and improvement.