Dataset Information

Reproducibility of Hospital Rankings Based on Centers for Medicare & Medicaid Services Hospital Compare Measures as a Function of Measure Reliability.

ABSTRACT:

Importance

Unreliable performance measures can mask poor-quality care and distort financial incentives in value-based purchasing.

Objective

To examine the association between test-retest reliability and the reproducibility of hospital rankings.

Design, setting, and participants

In a cross-sectional design, Centers for Medicare & Medicaid Services Hospital Compare data were analyzed for the 2017 (based on 2014-2017 data) and 2018 (based on 2015-2018 data) reporting periods. The study was conducted from December 13, 2020, to September 30, 2021. This analysis was based on 28 measures, including mortality (acute myocardial infarction, congestive heart failure, pneumonia, and coronary artery bypass grafting), readmissions (acute myocardial infarction, congestive heart failure, pneumonia, and coronary artery bypass grafting), and surgical complications (postoperative acute kidney failure, postoperative respiratory failure, postoperative sepsis, and failure to rescue).

Exposures

Measure reliability based on test-retest reliability testing.

Main outcomes and measures

The reproducibility of hospital rankings was quantified by calculating the reclassification rate across the 2017 and 2018 reporting periods after categorizing the hospitals into terciles, quartiles, deciles, and statistical outliers. Linear regression analysis was used to examine the association between the reclassification rate and the intraclass correlation coefficient for each of the classification systems.

Results

The analytic cohort consisted of 28 measures from 4452 hospitals with a median of 2927 (IQR, 2378-3160) hospitals contributing data for each measure. The hospitals participating in the Inpatient Prospective Payment System (n = 3195) had a median bed size of 141 (IQR, 69-261), average daily census of 70 (IQR, 24-155) patients, and a median disproportionate share hospital percentage of 38.2% (IQR, 18.7%-36.6%). The median intraclass correlation coefficient was 0.78 (IQR, 0.72-0.81), ranging between 0.50 and 0.85. The median reclassification rate was 70% (IQR, 62%-71%) when hospitals were ranked by deciles, 43% (IQR, 39%-45%) when ranked by quartiles, 34% (IQR, 31%-36%) when ranked by terciles, and 3.8% (IQR, 2.0%-6.2%) when ranked by outlier status. Increases in measure reliability were not associated with decreases in the reclassification rate. Each 0.1-point increase in the intraclass correlation coefficient was associated with a 6.80 (95% CI, 2.28-11.30; P = .005) percentage-point increase in the reclassification rate when hospitals were ranked into performance deciles, 4.15 (95% CI, 1.16-7.14; P = .008) when ranked into performance quartiles, 1.47 (95% CI, 1.84, 4.77; P = .37) when ranked into performance terciles, and 3.70 (95% CI, 1.30-6.09; P = .004) when ranked by outlier status.

Conclusions and relevance

In this study, more reliable measures were not associated with lower rates of reclassifying hospitals using test-retest reliability testing. These findings suggest that measure reliability should not be assessed with test-retest reliability testing.

SUBMITTER: Glance LG

PROVIDER: S-EPMC8652605 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:Importance:The Centers for Medicare & Medicaid Services (CMS) Five-Star measure for nursing homes is designed with input from expert panels about the importance of multiple quality indicators. Consumers may assign different values to these indicators, creating different 5-star ratings. Objective:To compare nursing homes' rankings based on the CMS Five-Star measure with rankings based on consumers' judgment about the importance of the same quality indicators. Design, Setting, and Participants:In this quality improvement study, CMS Five-Star data were linked with a measure calculated from CMS quality indicators and consumer values obtained from a national survey. Data covered the last quarter of 2016 and the first three quarters of 2017. The study included 10?676 nursing homes, comprising 69.8% of those with reported Five-Star measures. The national survey included adults, either nursing home residents or their family members who reported being familiar with the quality of care their relative received. Data analysis was performed from January 2019 to February 2020. Main Outcomes and Measures:The contingent valuation method was administered via the survey to obtain consumers' relative values of the quality indicators, and statistical analyses were used to create the contingent valuation measure. Agreement in nursing home rankings was assessed using the Five-Star measure, which is based on weights developed by expert panels, compared with rankings based on the contingent valuation measure. Results:Among 10?676 study nursing homes with a mean (SD) of 119.4 (59.4) beds, 7845 (73.5%) were for profit, 6424 (61.8%) were part of a chain, and 8009 (75.0%) were urban. The 4310 survey respondents (mean [SD] age, 39.9 [15.6] years; 1143 [26.5%] men; 3448 [80%] white) included mostly family members (3879 participants [90.0%]). The Pearson correlation coefficient (0.65) and weighted ? statistics (0.48) indicated only moderate agreement between ranking of nursing homes' performance by the 2 measures and disagreement on ranking for approximately one-half of the nursing homes. Conclusions and Relevance:Current nursing home report cards might not reflect consumers' values and the relative importance consumers place on each of the quality indicators that compose the overall Five-Star measure. Quality report cards might be more relevant to consumers by augmenting the Five-Star measure with a measure reflecting consumers' preferences. It is unknown whether these conclusions are generalizable to other report cards, such as Hospital and Home Health Compare, without conducting similar studies for these report cards.

Project description:BACKGROUND:Both the Centers for Medicare and Medicaid Services' (CMS) Hospital Compare star rating and surgical case volume have been publicized as metrics that can help patients to identify high-quality hospitals for complex care such as cancer surgery. The current study evaluates the relationship between the CMS' star rating, surgical volume, and short-term outcomes after major cancer surgery. METHODS:National Medicare data were used to evaluate the relationship between hospital star ratings and cancer surgery volume quintiles. Then, multilevel logistic regression models were fit to examine the association between cancer surgery outcomes and both star rankings and surgical volumes. Lastly, a graphical approach was used to compare how well star ratings and surgical volume predicted cancer surgery outcomes. RESULTS:This study identified 365,752 patients undergoing major cancer surgery for 1 of 9 cancer types at 2,550 hospitals. Star rating was not associated with surgical volume (P?<?.001). However, both the star rating and surgical volume were correlated with 4 short-term cancer surgery outcomes (mortality, complication rate, readmissions, and prolonged length of stay). The adjusted predicted probabilities for 5- and 1-star hospitals were 2.3% and 4.5% for mortality, 39% and 48% for complications, 10% and 15% for readmissions, and 8% and 16% for a prolonged length of stay, respectively. The adjusted predicted probabilities for hospitals with the highest and lowest quintile cancer surgery volumes were 2.7% and 5.8% for mortality, 41% and 55% for complications, 12.2% and 11.6% for readmissions, and 9.4% and 13% for a prolonged length of stay, respectively. Furthermore, surgical volume and the star rating were similarly associated with mortality and complications, whereas the star rating was more highly associated with readmissions and prolonged length of stay. CONCLUSIONS:In the absence of other information, these findings suggest that the star rating may be useful to patients when they are selecting a hospital for major cancer surgery. However, more research is needed before these ratings can supplant surgical volume as a measure of surgical quality. Cancer 2017;123:4259-4267. © 2017 American Cancer Society.

Project description:STUDY OBJECTIVES:Centers for Medicare and Medicaid Services (CMS) reimbursement for positive airway pressure (PAP) devices for obstructive sleep apnea treatment is dependent on patients meeting adherence expectations within the first 3 months on therapy. Adherence is defined as usage of the device for at least 4 hours per night on 70% of nights during a consecutive 30-day period. We hypothesize that the adherence pattern may be established beyond this initial period, which may limit the opportunity to treat many patients. METHODS:Treatment and adherence data from PAP devices were monitored via wireless modems for 42 consecutive PAP-naïve military veterans who completed 1 year of nightly monitoring. Their baseline characteristics were as follows: age (mean ± standard deviation) 58.5 ± 12.5 years; body mass index 33.7 ± 5.7 kg/m2; diagnostic apnea-hypopnea index (pretreatment) 28.1 ± 18.5 events/h; apnea-hypopnea index on PAP: 4.3 ± 3.3 events/h. We examined daily, monthly, quarterly, semiannual, and annual reports, and the best 30-day adherence report for each quarter. RESULTS:In the first 3 months, 19 of 42 participants were adherent by CMS criteria, and 23 of 42 participants were not. Of the 19 adherent participants, 13 remained adherent and 6 became nonadherent or stopped PAP treatment for the remainder of the year. In the 23 initially nonadherent participants, 16 stopped PAP treatment, and 7 participants (30.4%) became adherent (using CMS criteria) during the rest of the year. Thus, PAP adherence during the first 3 months was predictive for the rest of the year in only 68.4%. PAP nonadherence during the first 3 months was predictive for further nonadherence in only 69.6% of the cases. Overall, this led to a 65% sensitivity and 72% specificity of using adherence at 3 months in predicting adherence at 1 year. CONCLUSIONS:CMS adherence criteria affecting PAP coverage are restrictive and can result in the withholding of therapy in many patients who otherwise might become adherent. CLINICAL TRIAL REGISTRATION:Registry: ClinicalTrials.gov, Title: Remote Monitoring in Obstructive Sleep Apnea, Identifier: NCT01678560, URL: https:// clinicaltrials.gov/ct2/show/NCT01678560.

Project description:ImportancePredicting payments for particular conditions or populations is essential for research, benchmarking, public reporting, and calculations for population-based programs. Centers for Medicare & Medicaid Services (CMS) models often group codes into disease categories, but using single, rather than grouped, diagnostic codes and leveraging present on admission (POA) codes may enhance these models.ObjectiveTo determine whether changes to the candidate variables in CMS models would improve risk models predicting patient total payment within 30 days of hospitalization for acute myocardial infarction (AMI), heart failure (HF), and pneumonia.Design, setting, and participantsThis comparative effectiveness research study used data from Medicare fee-for-service hospitalizations for AMI, HF, and pneumonia at acute care hospitals from July 1, 2013, through September 30, 2015. Payments across multiple care settings, services, and supplies were included and adjusted for geographic and policy variations, corrected for inflation, and winsorized. The same data source was used but varied for the candidate variables and their selection, and the method used by CMS for public reporting that used grouped codes was compared with variations that used POA codes and single diagnostic codes. Combinations of use of POA codes, separation of index admission diagnoses from those in the previous 12 months, and use of individual International Classification of Diseases, Ninth Revision, Clinical Modification codes instead of grouped diagnostic categories were tested. Data analysis was performed from December 4, 2017, to June 10, 2019.Main outcomes and measuresThe models' goodness of fit was compared using root mean square error (RMSE) and the McFadden pseudo R2.ResultsAmong the 1 943 049 total hospitalizations of the study participants, 343 116 admissions were for AMI (52.5% male; 37.4% aged ≤74 years), 677 044 for HF (45.5% male; 25.9% aged ≤74 years), and 922 889 for pneumonia (46.4% male; 28.2% aged ≤74 years). The mean (SD) 30-day payment was $23 103 ($18 221) for AMI, $16 365 ($12 527) for HF, and $17 097 ($12 087) for pneumonia. Each incremental model change improved the pseudo R2 and RMSE. Incorporating all 3 changes improved the pseudo R2 of the patient-level models from 0.077 to 0.129 for AMI, from 0.042 to 0.129 for HF, and from 0.114 to 0.237 for pneumonia. Parallel improvements in RMSE were found for all 3 conditions.Conclusions and relevanceLeveraging POA codes, separating index from previous diagnoses, and using single diagnostic codes improved payment models. Better models can potentially improve research, benchmarking, public reporting, and calculations for population-based programs.