Dataset Information

Criteria for evaluating programme theory diagrams in quality improvement initiatives: a structured method for appraisal.

ABSTRACT: Despite criticisms that many quality improvement (QI) initiatives fail due to incomplete programme theory, there is no defined way to evaluate how programme theory has been articulated. The objective of this research was to develop, and assess the usability and reliability of scoring criteria to evaluate programme theory diagrams.Criteria development was informed by published literature and QI experts. Inter-rater reliability was tested between two evaluators. About 63 programme theory diagrams (42 driver diagrams and 21 action-effect diagrams) were reviewed to establish whether the criteria could support comparative analysis of different approaches to constructing diagrams.Components of the scoring criteria include: assessment of overall aim, logical overview, clarity of components, cause-effect relationships, evidence and measurement. Independent reviewers had 78% inter-rater reliability. Scoring enabled direct comparison of different approaches to developing programme theory; action-effect diagrams were found to have had a statistically significant but moderate improvement in programme theory quality over driver diagrams; no significant differences were observed based on the setting in which driver diagrams were developed.The scoring criteria summarise the necessary components of programme theory that are thought to contribute to successful QI projects. The viability of the scoring criteria for practical application was demonstrated. Future uses include assessment of individual programme theory diagrams and comparison of different approaches (e.g. methodological, teaching or other QI support) to produce programme theory. The criteria can be used as a tool to guide the production of better programme theory diagrams, and also highlights where additional support for QI teams could be needed.

SUBMITTER: Issen L

PROVIDER: S-EPMC6094797 | biostudies-other | 2018 Aug

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

Criteria for evaluating programme theory diagrams in quality improvement initiatives: a structured method for appraisal.

Issen Laurel L Woodcock Thomas T McNicholas Christopher C Lennox Laura L Reed Julie E JE

International journal for quality in health care : journal of the International Society for Quality in Health Care 20180801 7

<h4>Background</h4>Despite criticisms that many quality improvement (QI) initiatives fail due to incomplete programme theory, there is no defined way to evaluate how programme theory has been articulated. The objective of this research was to develop, and assess the usability and reliability of scoring criteria to evaluate programme theory diagrams.<h4>Methods</h4>Criteria development was informed by published literature and QI experts. Inter-rater reliability was tested between two evaluators. ...[more]

PMID: 29635294

Similar Datasets

Project description:ObjectiveValid, reliable critical appraisal tools advance quality improvement (QI) intervention impacts by helping stakeholders identify higher quality studies. QI approaches are diverse and differ from clinical interventions. Widely used critical appraisal instruments do not take unique QI features into account and existing QI tools (eg, Standards for QI Reporting Excellence) are intended for publication guidance rather than critical appraisal. This study developed and psychometrically tested a critical appraisal instrument, the QI Minimum Quality Criteria Set (QI-MQCS) for assessing QI-specific features of QI publications.MethodsApproaches to developing the tool and ensuring validity included a literature review, in-person and online survey expert panel input, and application to empirical examples. We investigated psychometric properties in a set of diverse QI publications (N=54) by analysing reliability measures and item endorsement rates and explored sources of disagreement between reviewers.ResultsThe QI-MQCS includes 16 content domains to evaluate QI intervention publications: Organisational Motivation, Intervention Rationale, Intervention Description, Organisational Characteristics, Implementation, Study Design, Comparator Description, Data Sources, Timing, Adherence/Fidelity, Health Outcomes, Organisational Readiness, Penetration/Reach, Sustainability, Spread and Limitations. Median inter-rater agreement for QI-MQCS items was κ 0.57 (83% agreement). Item statistics indicated sufficient ability to differentiate between publications (median quality criteria met 67%). Internal consistency measures indicated coherence without excessive conceptual overlap (absolute mean interitem correlation=0.19). The critical appraisal instrument is accompanied by a user manual detailing What to consider, Where to look and How to rate.ConclusionsWe developed a ready-to-use, valid and reliable critical appraisal instrument applicable to healthcare QI intervention publications, but recognise scope for continuing refinement.

Project description:BackgroundInstitutional Review Boards (IRBs) have been criticized for inconsistency, delay, and bias, suggesting an opportunity for quality improvement. To aid such quality improvement, this study aimed at determining IRB members' and investigators' priorities regarding IRB review at 10 Veterans Affairs (VA) IRBs.Methods680 anonymous internet surveys were sent to 252 IRB members and staff, and 428 principal investigators and project coordinators at 9 VA Medical Centers and the VA Central IRB. Surveys included 27 statements adapted from Koocher and Kieth-Spiegel's IRB-RAT describing IRB activities or functions (e.g., "An IRB that is open to reversing its earlier decisions"). Respondents indicated how each statement described both their "ideal" and "actual" IRBs. The difference between the ratings of the actual and ideal IRBs was calculated for each item along with estimated 95% confidence intervals. Ratings outside those intervals indicated activities or functions with relatively good or poor performance compared to the ideal IRB.Results390 (57.4%) responses from 165 IRB members and staff (65.5%) and 225 investigators and project coordinators (52.6%) demonstrated that these IRBs were closest to the ideal when protecting human subjects, treating investigators with respect, and taking appropriate action for alleged scientific misconduct. The IRBs were furthest from the ideal regarding duplicative forms, timeliness of review, and provision of complete rationales for decisions. Although IRB members reported near-ideal willingness to reverse earlier decisions, investigators rated this capacity far from ideal. Investigators rated IRB members' knowledge about procedures and policy as nearly ideal, but IRB members themselves rated this aspect far from ideal. Noteworthy site-level differences in the ratings of IRB functions and activities were also identified.ConclusionsAlthough these VA IRBs perform well in some areas, these data support the need for ongoing quality improvement. The described method of administering and analyzing the IRB-RAT may help identify and monitor site- and activity- specific initiatives for quality improvement.

Project description:BackgroundGuideline adherence in physical therapy is far from optimal, which has consequences for the effectiveness and efficiency of physical therapy care. Programmes to enhance guideline adherence have, so far, been relatively ineffective. We systematically developed a theory-based Quality Improvement in Physical Therapy (QUIP) programme aimed at the individual performance level (practicing physiotherapists; PTs) and the practice organization level (practice quality manager; PQM). The aim of the study was to pilot test the multilevel QUIP programme's effectiveness and the fidelity, acceptability and feasibility of its implementation.MethodsA one-group, pre-test, post-test pilot study (N?=?8 practices; N?=?32 PTs, 8 of whom were also PQMs) done between September and December 2009. Guideline adherence was measured using clinical vignettes that addressed 12 quality indicators reflecting the guidelines' main recommendations. Determinants of adherence were measured using quantitative methods (questionnaires). Delivery of the programme and management changes were assessed using qualitative methods (observations, group interviews, and document analyses). Changes in adherence and determinants were tested in the paired samples T-tests and expressed in effect sizes (Cohen's d).ResultsOverall adherence did not change (3.1%; p?=?.138). Adherence to three quality indicators improved (8%, 24%, 43%; .000 ? p ? .023). Adherence to one quality indicator decreased (-15.7%; p?=?.004). Scores on various determinants of individual performance improved and favourable changes at practice organizational level were observed. Improvements were associated with the programme's multilevel approach, collective goal setting, and the application of self-regulation; unfavourable findings with programme deficits. The one-group pre-test post-test design limits the internal validity of the study, the self-selected sample its external validity.ConclusionsThe QUIP programme has the potential to change physical therapy practice but needs considerable revision to induce the ongoing quality improvement process that is required to optimize overall guideline adherence. To assess its value, the programme needs to be tested in a randomized controlled trial.

Project description:ObjectiveTo evaluate the effectiveness of training programme on knowledge related to new interventions proposed under India Newborn Action Plan (INAP) and Integrated Action Plan against Pneumonia and Diarrhoea (IAPPD).DesignQuality improvement study with pre-evaluation and post evaluation.SettingThe study was conducted in 17 districts of Odisha, India.Participants and interventionsThe participants were healthcare providers and programme managers involved in maternal and child health programmes. Intervention was a short-structured (8 hours) training delivered to 127 batches with expected participation of 30 trainees in each batch. Training was divided into four modules covering new interventions related to INAP and IAPPD like causes of neonatal death, kangaroo mother care (KMC), feeding of low birthweight (LBW) infants, use of injection gentamicin, identification of possible serious bacterial infection (PSBI), identification and management of pneumonia and diarrhoea and key interventions for maternal health. Various modalities of teaching-learning method were used.Outcome measuresPretraining and post-training knowledge assessment was done with a pretested tool consisting of 15 items. Each item carried equal weightage in calculation of knowledge score thus maximum possible knowledge score was 15. Feedback assessment was also done after the training.ResultsThe mean (SD) knowledge score significantly improved to 10.24 (2.24) after training as compared with 4.73 (1.94) before training, p<0.001 (n=982). There was significant improvement in knowledge for majority of the components namely causes of neonatal death (61.9% vs 28.1%), KMC (68.0% vs 54.6%), feeding of LBW infants (77.7% vs 6.9%), use of injection gentamicin (69.7% vs 11.2%), identification of PSBI (69.5% vs 59.5%). The improvement in knowledge score was more when healthcare providers and programme managers had provided a favourable response on feedback.ConclusionSystematic pretest and post-test assessment coupled with feedback assessment can ensure the effectiveness of training programmes offered in programmatic settings.

Project description:BACKGROUND: Variation in effectiveness of continuous quality improvement (CQI) interventions between services is commonly reported, but with little explanation of how contextual and other factors may interact to produce this variation. Therefore, there is scant information available on which policy makers can draw to inform effective implementation in different settings. In this paper, we explore how patterns of change in delivery of services may have been achieved in a diverse range of health centers participating in a wide-scale program to achieve improvements in quality of care for Indigenous Australians. METHODS: We elicited key informants' interpretations of factors explaining patterns of change in delivery of guideline-scheduled services over three or more years of a wide-scale CQI project, and inductively analyzed these interpretations to propose fine-grained realist hypotheses about what works for whom and in what circumstances. Data were derived from annual clinical audits from 36 health centers operating in diverse settings, quarterly project monitoring reports, and workshops with 12 key informants who had key roles in project implementation. We abstracted potential context-mechanism-outcome configurations from the data, and based on these, identified potential program-strengthening strategies. RESULTS: Several context-specific, mechanism-based explanations for effectiveness of this CQI project were identified. These were collective valuing of clinical data for improvement purposes; collective efficacy; and organizational change towards a population health orientation. Health centers with strong central management of CQI, and those in which CQI efforts were more dependent on local health center initiative and were adapted to resonate with local priorities were both favorable contexts for collective valuing of clinical data. Where health centers had prior positive experiences of collaboration, effects appeared to be achieved at least partly through the mechanism of collective efficacy. Strong community linkages, staff ability to identify with patients, and staff having the skills and support to take broad ranging action, were favorable contexts for the mechanism of increased population health orientation. CONCLUSIONS: Our study provides evidence to support strategies for program strengthening described in the literature, and extends the understanding of mechanisms through which strategies may be effective in achieving particular outcomes in different contexts.

Dataset Information

Criteria for evaluating programme theory diagrams in quality improvement initiatives: a structured method for appraisal.

Publications

Criteria for evaluating programme theory diagrams in quality improvement initiatives: a structured method for appraisal.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets