Ranking hospitals on surgical mortality: the importance of reliability adjustment.
Ontology highlight
ABSTRACT: Objective
We examined the implications of reliability adjustment on hospital mortality with surgery.Data source
We used national Medicare data (2003-2006) for three surgical procedures: coronary artery bypass grafting (CABG), abdominal aortic aneurysm (AAA) repair, and pancreatic resection.Study design
We conducted an observational study to evaluate the impact of reliability adjustment on hospital mortality rankings. Using hierarchical modeling, we adjusted hospital mortality for reliability using empirical Bayes techniques. We assessed the implication of this adjustment on the apparent variation across hospitals and the ability of historical hospital mortality rates (2003-2004) to forecast future mortality (2005-2006).Principal findings
The net effect of reliability adjustment was to greatly diminish apparent variation for all three operations. Reliability adjustment was also particularly important for identifying hospitals with the lowest future mortality. Without reliability adjustment, hospitals in the "best" quintile (2003-2004) with pancreatic resection had a mortality of 7.6 percent in 2005-2006; with reliability adjustment, the "best" hospital quintile had a mortality of 2.7 percent in 2005-2006. For AAA repair, reliability adjustment also improved the ability to identify hospitals with lower future mortality. For CABG, the benefits of reliability adjustment were limited to the lowest volume hospitals.Conclusion
Reliability adjustment results in more stable estimates of mortality that better forecast future performance. This statistical technique is crucial for helping patients select the best hospitals for specific procedures, particularly uncommon ones, and should be used for public reporting of hospital mortality.
SUBMITTER: Dimick JB
PROVIDER: S-EPMC2976775 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BACKGROUND:The conceptualization of hospital quality indicators usually includes some form of risk adjustment to account for hospital differences in case mix. For binary outcome variables like in-hospital mortality, frequently utilized risk adjusted measures include the standardized mortality ratio (SMR), the risk standardized mortality rate (RSMR), and excess risk (ER). All of these measures require the estimation of expected hospital mortality, which is often based on logistic regression models. In this context, an issue that is often neglected is correlation between hospital performance (e.g. care quality) and patient-specific risk factors. The objective of this study was to investigate the impact of such correlation on the adequacy of hospital rankings based on different measures and methods. METHODS:Using Monte Carlo simulation, the impact of correlation between hospital care quality and patient-specific risk factors on the adequacy of hospital rankings was assessed for SMR/RSMR, and ER based on logistic regression and random effects logistic regression. As an alternative method, fixed effects logistic regression with Firth correction was considered. The adequacies of the resulting hospital rankings were assessed by the shares of hospitals correctly classified into quintiles according to their true (unobserved) care qualities. RESULTS:The performance of risk adjustment approaches based on logistic regression and random effects logistic regression declined when correlation between care quality and a risk factor was induced. In contrast, fixed-effects-based estimations proved to be more robust. This was particularly true for fixed-effects-logistic-regression-based ER. In the absence of correlation between risk factors and care quality, all approaches showed similar performance. CONCLUSIONS:Correlation between risk factors and hospital performance may severely bias hospital rankings based on logistic regression and random effects logistic regression. ER based on fixed effects logistic regression with Firth correction should be considered as an alternative approach to assess hospital performance.
| S-EPMC6892499 | biostudies-literature
Project description:BackgroundThe Surgical Outcome Risk Tool (SORT) was derived and validated in the UK to improve preoperative prediction of postoperative risk. The aim of this study was to validate the SORT in a European mixed-case surgical population outside of the UK.MethodsThe study included patients aged at least 18 years with ASA Physical Status (ASA-PS) grades I-V who underwent non-cardiac surgery at four tertiary hospitals in Sweden between November 2015 and February 2016. Exclusion criteria were surgery under local anaesthesia and missing data on the SORT predictors (ASA-PS, surgical urgency, high-risk surgery, surgical severity, malignancy, age over 65 years). The outcome was 30-day mortality. Discrimination and calibration of the SORT were assessed using area under the receiver operating curve (AUROC) statistics and calibration plots. A sensitivity analysis was done in a high-risk subgroup (ASA-PS III or higher; surgical complexity major to Xmajor according to the SORT; gastrointestinal, orthopaedic, urogenital/obstetric surgery; and age at least 18 years).ResultsThe validation cohort included 17 965 patients; median age was 58 (i.q.r. 40-70) years, 43.2 per cent were men, and the mortality rate at 30 days was 1.6 per cent. The SORT had excellent discrimination, with an AUROC of 0.91 (95 per cent c.i. 0.89 to 0.92), and good calibration. The high-risk subgroup (1807 patients) had a 30-day mortality rate of 5.6 per cent; in the sensitivity analysis, the SORT had good discrimination, with an AUROC of 0.79 (0.74 to 0.83), and calibration remained good.ConclusionThe estimates of the original the SORT for prediction of 30-day mortality were valid and reliable in a mixed-case surgical population in a non-UK European setting.
| S-EPMC10364531 | biostudies-literature
Project description:OBJECTIVES:To extend reliability of WHO Behaviourally Anchored Rating Scale (WHOBARS) to measure the quality of WHO Surgical Safety Checklist administration using generalisability theory. In this context, extending reliability refers to establishing generalisability of the tool scores across populations of teams and raters by accounting for the relevant sources of measurement errors. DESIGN:Cross-sectional random effect measurement design assessing surgical teams by the five items on the three Checklist phases, and at three sites by two trained raters simultaneously. SETTING:The data were collected in three tertiary hospitals in Auckland, New Zealand in 2016 and included 60 teams observed in 60 different cases with an equal number of teams (n=20) per site. All elective and acute cases (adults and children) involving surgery under general anaesthesia during normal working hours were eligible. PARTICIPANTS:The study included 243 surgical staff members, 138 (50.12%) women. MAIN OUTCOME MEASURE:Absolute generalisability coefficient that accounts for variance due to items, phases, sites and raters for the WHOBARS measure of the quality of WHO Surgical Safety Checklist administration. RESULTS:The WHOBARS in its present form has demonstrated good generalisability of scores across teams and raters (G absolute=0.83). The largest source of measurement error was the interaction between the surgical team and the rater, accounting for 16.7% (95% CI 16.4 to 16.9) of the total variance in the data. Removing any items from the WHOBARS led to a decrease in the overall reliability of the instrument. CONCLUSIONS:Assessing checklist administration quality is important for promoting improvement in its use, and WHOBARS offers a reliable approach for doing this.
| S-EPMC6340010 | biostudies-literature
Project description:International financial organisations like the International Monetary Fund (IMF) play a central role in shaping the developmental trajectories of fiscally distressed countries through their conditional lending schemes, known as 'structural adjustment programmes'. These programmes entail wide-ranging domestic policy reforms that influence local health and welfare systems. Using novel panel data from 187 countries between 1990 and 2017 and an instrumental variable technique, we find that IMF programmes lead to over 70 excess deaths from respiratory diseases and tuberculosis per 100,000 population and that IMF-mandated privatisation reforms lead to over 90 excess deaths per 100,000 population. Thus structural adjustment programmes, as currently designed and implemented, are harmful to population health and increase global infectious disease burdens.
| S-EPMC9286264 | biostudies-literature
Project description:BackgroundThe Centers for Medicare and Medicaid Services (CMS) use colon surgical site infection (SSI) rates to rank hospitals and apply financial penalties. The CMS' risk-adjustment model omits potentially impactful variables that might disadvantage hospitals with complex surgical populations.MethodsWe analyzed adult patients who underwent colon surgery within facilities associated with HCA Healthcare from 2014 to 2016. SSIs were identified from National Health Safety Network (NHSN) reporting. We trained and validated 3 SSI prediction models, using (1) current CMS model variables, including hospital-specific random effects (HCA-adapted CMS model); (2) demographics and claims-based comorbidities (expanded-claims model); and (3) demographics, claims-based comorbidities, and NHSN variables (claims-plus-electronic health record [EHR] model). Discrimination, calibration, and resulting rankings were compared among all models and the current CMS model with published coefficient values.ResultsWe identified 39 468 colon surgeries in 149 hospitals, resulting in 1216 (3.1%) SSIs. Compared to the HCA-adapted CMS model, the expanded-claims model had similar performance (c-statistic, 0.65 vs 0.67, respectively), while the claims-plus-EHR model was more accurate (c-statistic, 0.70; 95% confidence interval, .67-.73; P = .004). The sampling variation, due to the low surgical volume and small number of infections, contributed 74% of the total variation in observed SSI rates between hospitals. When CMS model rankings were compared to those from the expanded-claims and claims-plus-EHR models, 18 (15%) and 26 (22%) hospitals changed quartiles, respectively, and 10 (8.3%) and 12 (10%) hospitals changed into or out of the lowest-performing quartile, respectively.ConclusionsAn expanded set of variables improved colon SSI risk predictions and quartile assignments, but low procedure volumes and SSI events remain a barrier to effectively comparing hospitals.
| S-EPMC7823072 | biostudies-literature
Project description:Globally sclerotic glomeruli (GSG) occur with both normal aging and kidney disease. However, it is unknown whether any GSG or only GSG exceeding that expected for age is clinically important. To evaluate this, we identified patients with a glomerulopathy that often presents with nephrotic syndrome (focal segmental glomerulosclerosis, membranous nephropathy, or minimal change disease) in the setting of the Nephrotic Syndrome Study Network (NEPTUNE), China-Digital Kidney Pathology (DiKiP), and the Southeast Minnesota cohorts. Age-based thresholds (95th percentile) for GSG based on normotensive living kidney donors were used to classify each patient into one of three groups; no GSG, GSG normal for age, or GSG abnormal for age. The risk of end-stage renal disease or a 40% decline in glomerular filtration rate during follow-up was then compared between groups. Among the 425 patients studied, 170 had no GSG, 107 had GSG normal for age, and 148 had GSG abnormal for age. Compared to those with no GSG, the risk of kidney disease progression with GSG normal for age was similar but was significantly higher with GSG abnormal for age. This increased risk with GSG abnormal for age remained significant after adjustment for interstitial fibrosis, arteriosclerosis, age, hypertension, diabetes, body mass index, glomerulopathy type, glomerular filtration rate, and proteinuria. Thus, in patients with glomerulopathy that often presents with nephrotic syndrome, global glomerulosclerosis is clinically important only if it exceeds that expected for age.
| S-EPMC5911429 | biostudies-literature
Project description:There is no decision-making framework in the early-adoption stage of novel surgical technologies, putting the quality of health care and resource allocation of the health care system at risk. To investigate relevant weighted criteria that decision-makers may use to make an informed decision for the early adoption of innovative surgical technologies. This multi-institutional decision analytical modeling study used a mixed-methods multicriteria decision analysis (MCDA) and had 2 phases. First, a panel of 12 experts validated decision criteria in the literature and identified additional criteria. Second, 33 Canadian experts prioritized the main criteria (domains) using the composition pairwise-comparison weight-elicitation method (analytical hierarchy process model) and ranked their subcriteria using the direct-ranking elicitation method (Likert scale). Data were analyzed, and response consistency was estimated using the consistency ratio. Analysis of variance was used to assess for significant differences between expert responses. The MCDA was conducted at McGill University between 2021 and 2023. Data were collected nationally by inviting experts in Canada. Criteria domain weights and subcriteria rankings. Priority vectors, which are priority scores analyzed and prioritized from expert responses, were used to rank criteria domains and subcriteria for decision-making on adopting new innovative surgical technologies. A total of 45 experts (33 male [73.3%] and 12 female [26.7%]) were invited with different levels of education (22 experts with MD or equivalent, 13 experts with master's degree, and 12 experts with PhD degree) and years of experience (4 experts with <10, 12 experts with 11-20, 18 experts with 21-30, and 11 experts with >30 years). Surgeon experts (23 individuals) were from all surgical disciplines, and nonsurgeon experts (22 individuals) were administrative officers in surgical device procurement, health technology assessment experts, and hospital directors. A total of 7 domains and 44 subcriteria were identified. The MCDA model found that clinical outcomes had the highest priority vector, at 0.429, followed by patients and public relevance (0.135). Hospital-specific criteria (priority vector, 0.099), technology-specific criteria (priority vector, 0.092), and physician-specific criteria (priority vector, 0.087) were the next most highly ranked. The lowest priority vectors were for economic criteria, at 0.083, and finally policies and procedures, at 0.075. There was consensus in the responses (consistency ratio = 0.006), and there were no statistically significant differences between expert responses. This study weighted priority criteria domains in importance and established ranked subcriteria for decision-making of early adoption of surgical technologies. Putting these criteria into a framework may help surgeons and decision-makers make informed decisions for the early adoption of new surgical technologies.
| S-EPMC10654796 | biostudies-literature
Project description:ObjectiveDespite a growing volume of surgical procedures in low-income and middle-income countries, the costs of these procedures are not well understood. We estimated the costs of 12 surgical procedures commonly conducted in five different types of hospitals in India from the provider perspective, using a microcosting method.DesignCost and utilisation data were collected retrospectively from April 2010 to March 2011 to avoid seasonal variability.SettingFor this study, we chose five hospitals of different types: a 57-bed charitable hospital, a 200-bed private hospital, a 400-bed district hospital, a 655-bed private teaching hospital and a 778-bed tertiary care teaching hospital based on their willingness to cooperate and data accessibility. The hospitals were from four states in India. The private, charitable and tertiary care hospitals serve urban populations, the district hospital serves a semiurban area and the private teaching hospital serves a rural population.ResultsCosts of conducting lower section caesarean section ranged from rupees 2469 to 41 087; hysterectomy rupees 4124 to 57 622 and appendectomy rupees 2421 to 3616 (US$1=rupees 52). We computed the costs of conducting lap and open cholecystectomy (rupees 27 732 and 44 142, respectively); hernia repair (rupees 13 204); external fixation (rupees 8406); intestinal obstruction (rupees 6406); amputation (rupees 5158); coronary artery bypass graft (rupees 177 141); craniotomy (rupees 75 982) and functional endoscopic sinus surgery (rupees 53 398).ConclusionsEstimated costs are roughly comparable with rates of reimbursement provided by the Rashtriya Swasthya Bima Yojana (RSBY)-India's government-financed health insurance scheme that covers 32.4 million poor families. Results from this type of study can be used to set and revise the reimbursement rates.
| S-EPMC3686227 | biostudies-literature
Project description:BackgroundAlthough there is evidence that hospitals recognized for nursing excellence--Magnet hospitals--are successful in attracting and retaining nurses, it is uncertain whether Magnet recognition is associated with better patient outcomes than non-Magnets, and if so why.ObjectivesTo determine whether Magnet hospitals have lower risk-adjusted mortality and failure-to-rescue compared with non-Magnet hospitals, and to determine the most likely explanations. METHOD AND STUDY DESIGN: Analysis of linked patient, nurse, and hospital data on 56 Magnet and 508 non-Magnet hospitals. Logistic regression models were used to estimate differences in the odds of mortality and failure-to-rescue for surgical patients treated in Magnet versus non-Magnet hospitals, and to determine the extent to which differences in outcomes can be explained by nursing after accounting for patient and hospital differences.ResultsMagnet hospitals had significantly better work environments and higher proportions of nurses with bachelor's degrees and specialty certification. These nursing factors explained much of the Magnet hospital effect on patient outcomes. However, patients treated in Magnet hospitals had 14% lower odds of mortality (odds ratio 0.86; 95% confidence interval, 0.76-0.98; P=0.02) and 12% lower odds of failure-to-rescue (odds ratio 0.88; 95% confidence interval, 0.77-1.01; P=0.07) while controlling for nursing factors as well as hospital and patient differences.ConclusionsThe lower mortality we find in Magnet hospitals is largely attributable to measured nursing characteristics but there is a mortality advantage above and beyond what we could measure. Magnet recognition identifies existing quality and stimulates further positive organizational behavior that improves patient outcomes.
| S-EPMC3568449 | biostudies-literature
Project description:BackgroundThe US News and World Report reputation score correlates strongly with overall rank in adult and pediatric hospital rankings. Social media affects how information is disseminated to physicians and is used by hospitals as a marketing tool to recruit patients. It is unclear whether the reputation score for adult and children's hospitals relates to social media presence.ObjectiveThe objective of our study was to analyze the association between a hospital's social media metrics and the US News 2017-2018 Best Hospital Rankings for adult and children's hospitals.MethodsWe conducted a cross-sectional analysis of the reputation score, total score, and social media metrics (Twitter, Facebook, and Instagram) of hospitals who received at least one subspecialty ranking in the 2017-2018 US News publicly available annual rankings. Regression analysis was employed to analyze the partial correlation coefficients between social media metrics and a hospital's total points (ie, rank) and reputation score for both adult and children's hospitals while controlling for the bed size and time on Twitter.ResultsWe observed significant correlations for children's hospitals' reputation score and total points with the number of Twitter followers (total points: r=.465, P<.001; reputation: r=.524, P<.001) and Facebook followers (total points: r=.392, P=.002; reputation: r=.518, P<.001). Significant correlations for the adult hospitals reputation score were found with the number of Twitter followers (r=.848, P<.001), number of tweets (r=.535, P<.001), Klout score (r=.242, P=.02), and Facebook followers (r=.743, P<.001). In addition, significant correlations for adult hospitals total points were found with Twitter followers (r=.548, P<.001), number of tweets (r=.358, P<.001), Klout score (r=.203, P=.05), Facebook followers (r=.500, P<.001), and Instagram followers (r=.692, P<.001).ConclusionsA statistically significant correlation exists between multiple social media metrics and both a hospital's reputation score and total points (ie, overall rank). This association may indicate that a hospital's reputation may be influenced by its social media presence or that the reputation or rank of a hospital drives social media followers.
| S-EPMC6250955 | biostudies-literature