Dataset Information

Should policy-makers and managers trust PSI? An empirical validation study of five patient safety indicators in a national health service.

ABSTRACT:

Background

Patient Safety Indicators (PSI) are being modestly used in Spain, somewhat due to concerns on their empirical properties. This paper provides evidence by answering three questions: a) Are PSI differences across hospitals systematic -rather than random?; b) Do PSI measure differences among hospital-providers -as opposed to differences among patients?; and, c) Are measurements able to detect hospitals with a higher than "expected" number of cases?

Methods

An empirical validation study on administrative data was carried out. All 2005 and 2006 publicly-funded hospital discharges were used to retrieve eligible cases of five PSI: Death in low-mortality DRGs (MLM); decubitus ulcer (DU); postoperative pulmonary embolism or deep-vein thrombosis (PE-DVT); catheter-related infections (CRI), and postoperative sepsis (PS). Empirical Bayes statistic (EB) was used to estimate whether the variation was systematic; logistic-multilevel modelling determined what proportion of the variation was explained by the hospital; and, shrunken residuals, as provided by multilevel modelling, were plotted to flag hospitals performing worse than expected.

Results

Variation across hospitals was observed to be systematic in all indicators, with EB values ranging from 0.19 (CI95%:0.12 to 0.28) in PE-DVT to 0.34 (CI95%:0.25 to 0.45) in DU. A significant proportion of the variance was explained by the hospital, once patient case-mix was adjusted: from a 6% in MLM (CI95%:3% to 11%) to a 24% (CI95%:20% to 30%) in CRI. All PSI were able to flag hospitals with rates over the expected, although this capacity decreased when the largest hospitals were analysed.

Conclusion

Five PSI showed reasonable empirical properties to screen healthcare performance in Spanish hospitals, particularly in the largest ones.

SUBMITTER: Bernal-Delgado E

PROVIDER: S-EPMC3350386 | biostudies-literature | 2012 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Should policy-makers and managers trust PSI? An empirical validation study of five patient safety indicators in a national health service.

Bernal-Delgado Enrique E García-Armesto Sandra S Martínez-Lizaga Natalia N Abadía-Taira Begoña B Beltrán-Peribañez Joaquín J Peiró Salvador S

BMC medical research methodology 20120227

<h4>Background</h4>Patient Safety Indicators (PSI) are being modestly used in Spain, somewhat due to concerns on their empirical properties. This paper provides evidence by answering three questions: a) Are PSI differences across hospitals systematic -rather than random?; b) Do PSI measure differences among hospital-providers -as opposed to differences among patients?; and, c) Are measurements able to detect hospitals with a higher than "expected" number of cases?<h4>Methods</h4>An empirical val ...[more]

PMID: 22369291

Similar Datasets

Project description:BACKGROUND:It is unclear how to engage a wide range of knowledge users in research. We aimed to map the evidence on engaging knowledge users with an emphasis on policy-makers, health system managers, and policy analysts in the knowledge synthesis process through a scoping review. METHODS:We used the Joanna Briggs Institute guidance for scoping reviews. Nine electronic databases (e.g., MEDLINE), two grey literature sources (e.g., OpenSIGLE), and reference lists of relevant systematic reviews were searched from 1996 to August 2016. We included any type of study describing strategies, barriers and facilitators, or assessing the impact of engaging policy-makers, health system managers, and policy analysts in the knowledge synthesis process. Screening and data abstraction were conducted by two reviewers independently with a third reviewer resolving discrepancies. Frequency and thematic analyses were conducted. RESULTS:After screening 8395 titles and abstracts followed by 394 full-texts, 84 unique documents and 7 companion reports fulfilled our eligibility criteria. All 84 documents were published in the last 10 years, and half were prepared in North America. The most common type of knowledge synthesis with knowledge user engagement was a systematic review (36%). The knowledge synthesis most commonly addressed an issue at the level of national healthcare system (48%) and focused on health services delivery (17%) in high-income countries (86%). Policy-makers were the most common (64%) knowledge users, followed by healthcare professionals (49%) and government agencies as well as patients and caregivers (34%). Knowledge users were engaged in conceptualization and design (49%), literature search and data collection (52%), data synthesis and interpretation (71%), and knowledge dissemination and application (44%). Knowledge users were most commonly engaged as key informants through meetings and workshops as well as surveys, focus groups, and interviews either in-person or by telephone and emails. Knowledge user content expertise/awareness was a common facilitator (18%), while lack of time or opportunity to participate was a common barrier (12%). CONCLUSIONS:Knowledge users were most commonly engaged during the data synthesis and interpretation phases of the knowledge synthesis conduct. Researchers should document and evaluate knowledge user engagement in knowledge synthesis. REGISTRATION DETAILS:Open Science Framework ( https://osf.io/4dy53/ ).

Project description:BackgroundThe use of research evidence as an input for health decision-making is a need for most health systems. There are a number of approaches for promoting evidence use at different levels of the health system, but knowledge of their effectiveness is still scarce. The objective of this overview was to evaluate the effectiveness of knowledge communication and dissemination interventions, strategies or approaches targeting policy-makers and health managers.MethodsThis overview of systematic reviews used systematic review methods and was conducted according to a predefined and published protocol. A comprehensive electronic search of 13 databases and a manual search in four websites were conducted. Both published and unpublished reviews in English, Spanish or Portuguese were included. A narrative synthesis was undertaken, and effectiveness statements were developed, informed by the evidence identified.ResultsWe included 27 systematic reviews. Three studies included only a communication strategy, while eight only included dissemination strategies, and the remaining 16 included both. None of the selected reviews provided "sufficient evidence" for any of the strategies, while four provided some evidence for three communication and four dissemination strategies. Regarding communication strategies, the use of tailored and targeted messages seemed to successfully lead to changes in the decision-making practices of the target audience. Regarding dissemination strategies, interventions that aimed at improving only the reach of evidence did not have an impact on its use in decisions, while interventions aimed at enhancing users' ability to use and apply evidence had a positive effect on decision-making processes. Multifaceted dissemination strategies also demonstrated the potential for changing knowledge about evidence but not its implementation in decision-making.ConclusionsThere is limited evidence regarding the effectiveness of interventions targeting health managers and policy-makers, as well as the mechanisms required for achieving impact. More studies are needed that are informed by theoretical frameworks or specific tools and using robust methods, standardized outcome measures and clear descriptions of the interventions. We found that passive communication increased access to evidence but had no effect on uptake. Some evidence indicated that the use of targeted messages, knowledge-brokering and user training was effective in promoting evidence use by managers and policy-makers.

Project description:There is increasing interest globally in the use of more rigorous processes to ensure that maternal, newborn, and child health (MNCH) care recommendations are informed by the best available research evidence use. The purpose of this study was to engage Nigerian MNCH policy-makers and other stakeholders to consider issues around research to policy and practice interface and to assess their existing knowledge and capacity on the use of research evidence for policy-making and practice.The study design is a cross-sectional evaluation of MNCH stakeholders' knowledge as it pertains different dimensions of research to practice. This was undertaken during a national MNCH stakeholders' engagement event convened under the auspices of the West African Health Organization (WAHO) and the Federal Ministry of Health (FMoH) in Abuja, Nigeria. A questionnaire was administered to participants, which was designed to assess participants' knowledge, capacity and organizational process of generation, synthesis and utilization of research evidence in policy-making regarding MNCH.A total of 40 participants signed the informed consent form and completed the questionnaire. The mean ratings (MNRs) of participants' knowledge of electronic databases and capacity to identify and obtain relevant research evidence from electronic databases ranged from 3.62-3.68 on the scale of 5. The MNRs of participants' level of understanding of a policy brief, a policy dialogue and the role of researchers in policy-making ranged from 3.50-3.86. The MNRs of participants' level of understanding of evidence in policy-making context, types and sources of evidence, capacity to identify, select, adapt, and transform relevant evidence into policy ranged from 3.63-4.08. The MNRs of the participants' organization's capacity to cover their geographical areas of operation were generally low ranging from 3.32-3.38 in terms of manpower, logistics, facilities, and external support. The lowest MNR of 2.66 was recorded in funding.The outcomes of this study suggest that a stakeholders' engagement event can serve as an important platform to assess policy-makers' knowledge and capacity for evidence-informed policy-making and for the promotion of evidence use in the policy process.

Project description:BackgroundSystematic reviews are infrequently used by health care managers (HCMs) and policy-makers (PMs) in decision-making. HCMs and PMs co-developed and tested novel systematic review of effects formats to increase their use.MethodsA three-phased approach was used to evaluate the determinants to uptake of systematic reviews of effects and the usability of an innovative and a traditional systematic review of effects format. In phase 1, survey and interviews were conducted with HCMs and PMs in four Canadian provinces to determine perceptions of a traditional systematic review format. In phase 2, systematic review format prototypes were created by HCMs and PMs via Conceptboard©. In phase 3, prototypes underwent usability testing by HCMs and PMs.ResultsTwo hundred two participants (80 HCMs, 122 PMs) completed the phase 1 survey. Respondents reported that inadequate format (Mdn?=?4; IQR?=?4; range?=?1-7) and content (Mdn?=?4; IQR?=?3; range?=?1-7) influenced their use of systematic reviews. Most respondents (76%; n?=?136/180) reported they would be more likely to use systematic reviews if the format was modified. Findings from 11 interviews (5 HCMs, 6 PMs) revealed that participants preferred systematic reviews of effects that were easy to access and read and provided more information on intervention effectiveness and less information on review methodology. The mean System Usability Scale (SUS) score was 55.7 (standard deviation [SD] 17.2) for the traditional format; a SUS score?<?68 is below average usability. In phase 2, 14 HCMs and 20 PMs co-created prototypes, one for HCMs and one for PMs. HCMs preferred a traditional information order (i.e., methods, study flow diagram, forest plots) whereas PMs preferred an alternative order (i.e., background and key messages on one page; methods and limitations on another). In phase 3, the prototypes underwent usability testing with 5 HCMs and 7 PMs, 11 out of 12 participants co-created the prototypes (mean SUS score 86 [SD 9.3]).ConclusionsHCMs and PMs co-created prototypes for systematic review of effects formats based on their needs. The prototypes will be compared to a traditional format in a randomized trial.

Project description:BackgroundRapid reviews are increasingly used by policy agencies to access relevant research in short timeframes. Despite the growing number of programmes, little is known about how rapid reviews are used by health policy agencies. This study examined whether and how rapid reviews commissioned using a knowledge brokering programme were used by Australian policy-makers.MethodsThis study used interview data to examine the use of 139 rapid reviews by health policy agencies that were commissioned between 2006 and 2015. Transcripts were coded to identify how rapid reviews were used, the type of policy processes in which they were used, what evidence of use was provided and what reasons were given when rapid reviews were not used. Fisher's exact test was used to assess variation between types of agencies.ResultsOverall, 89% of commissioned rapid reviews were used by the commissioning agencies and 338 separate instances of use were identified, namely, on average, three uses per review. Policy-makers used reviews primarily to determine the details of a policy or programme, identify priorities for future action or investment, negotiate interjurisdictional decisions, evaluate alternative solutions for a policy problem, and communicate information to stakeholders. Some variation in use was observed across agencies. Reasons for non-use were related to changes in organisational structures, resources or key personnel in the commissioning agencies, or changes in the broader political environment.ConclusionsThis study found that almost all rapid reviews had been used by the agencies who commissioned them, primarily in policy and programme development, agenda-setting, and to communicate information to stakeholders. Reviews were used mostly in instrumental and conceptual ways and there was little evidence of symbolic use. Variations in use were identified across agencies. The findings suggest that commissioned rapid reviews are an effective means of providing timely relevant research for use in policy processes and that review findings may be applied in a variety of ways.

Dataset Information

Should policy-makers and managers trust PSI? An empirical validation study of five patient safety indicators in a national health service.

Background

Methods

Results

Conclusion

Publications

Should policy-makers and managers trust PSI? An empirical validation study of five patient safety indicators in a national health service.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure