Performance Characteristics of Profiling Methods and the Impact of Inadequate Case-mix Adjustment.
Ontology highlight
ABSTRACT:
SUBMITTER: Chen Y
PROVIDER: S-EPMC7731974 | biostudies-literature | 2019
REPOSITORIES: biostudies-literature
Publications
Performance Characteristics of Profiling Methods and the Impact of Inadequate Case-mix Adjustment.
Chen Yanjun Y Şentürk Damla D Estes Jason P JP Campos Luis F LF Rhee Connie M CM Dalrymple Lorien S LS Kalantar-Zadeh Kamyar K Nguyen Danh V DV
Communications in statistics: Simulation and computation 20190328
Profiling or evaluation of health care providers involves the application of statistical models to compare each provider's performance with respect to a patient outcome, such as unplanned 30-day hospital readmission, adjusted for patient case-mix characteristics. The nationally adopted method is based on random effects (RE) hierarchical logistic regression models. Although RE models are sensible for modeling hierarchical data, novel high dimensional fixed effects (FE) models have been proposed w ...[more]
Similar Datasets
Project description:ObjectiveTo assess the effect of case-mix adjustment on community health center (CHC) performance on patient experience measures.Data sourcesA Medicaid-managed care plan in Washington State collected patient survey data from 33 CHCs over three fiscal quarters during 2007-2008. The survey included three composite patient experience measures (6-month reports) and two overall ratings of care. The analytic sample includes 2,247 adult patients and 2,859 adults reporting for child patients.Study designWe compared the relative importance of patient case-mix adjusters by calculating each adjuster's predictive power and variability across CHCs. We then evaluated the impact of case-mix adjustment on the relative ranking of CHCs.Principal findingsImportant case-mix adjusters included adult self-reported health status or parent-reported child health status, adult age, and educational attainment. The effects of case-mix adjustment on patient reports and ratings were different in the adult and child samples. Adjusting for race/ethnicity and language had a greater impact on parent reports than adult reports, but it impacted ratings similarly across the samples. The impact of adjustment on composites and ratings was modest, but it affected the relative ranking of CHCs.ConclusionsTo ensure equitable comparison of CHC performance on patient experience measures, reports and ratings should be adjusted for adult self-reported health status or parent-reported child health status, adult age, education, race/ethnicity, and survey language. Because of the differential impact of case-mix adjusters for child and adult surveys, initiatives should consider measuring and reporting adult and child scores separately.
| S-EPMC2875754 | biostudies-literature
Project description:Quantitative metrics are used to develop profiles of health care institutions, including hospitals, nursing homes, and dialysis clinics. These profiles serve as measures of quality of care, which are used to compare institutions and determine reimbursement, as a part of a national effort led by the Center for Medicare and Medicaid Services in the United States. However, there is some concern about how misclassification in case-mix factors, which are typically accounted for in profiling, impacts results. We evaluated the potential effect of misclassification on profiling results, using 20 744 patients from 2740 dialysis facilities in the US Renal Data System. In this case study, we compared 30-day readmission as the profiling outcome measure, using comorbidity data from either the Center for Medicare and Medicaid Services Medical Evidence Report (error-prone) or Medicare claims (more accurate). Although the regression coefficient of the error-prone covariate demonstrated notable bias in simulation, the outcome measure-standardized readmission ratio-and profiling results were quite robust; for example, correlation coefficient of 0.99 in standardized readmission ratio estimates. Thus, we conclude that misclassification on case-mix did not meaningfully impact overall profiling results. We also identified both extreme degree of case-mix factor misclassification and magnitude of between-provider variability as 2 factors that can potentially exert enough influence on profile status to move a clinic from one performance category to another (eg, normal to worse performer).
| S-EPMC7265077 | biostudies-literature
Project description:BackgroundTo evaluate the performance of the patient clinical complexity level (PCCL) mechanism, which is the patient-level complexity adjustment factor within the Korean Diagnosis-Related Groups (KDRG) patient classification system, in explaining the variation in resource consumption within age adjacent diagnosis-related groups (AADRGs).MethodsWe used the inpatient claims data from a public hospital in Korea from 1 January 2017 to 30 June 2019, with 18 846 claims and 138 AADRGs. The differences in the total average payment between the four PCCL levels for each AADRG was tested using ANOVA and Duncan's post hoc test. The three patterns of differences with R-squared were as follows: the PCCL reflected the complexity well (valid); the average payment for PCCL 2, 3, and 4 was greater than PCCL 0 (partially valid); the PCCL did not reflect the complexity (not valid).ResultsThere were 9 (6.52%), 26 (18.84%), and 103 (74.64%) ADRGs included in the valid, partially valid, and not valid categories, respectively. The average R-squared values were 32.18, 40.81, and 35.41%, respectively, with an average R-squared for all patterns of 36.21%.ConclusionsAdjustment using the PCCL in the KDRG classification system exhibited low performance in explaining the variation in resource consumption within AADRGs. As the KDRG classification system is used for reimbursement under the new DRG-based prospective payment system (PPS) pilot project, with plans for expansion, there should be an overall review of the validity of the complexity and rationality of using the KDRG classification system.
| S-EPMC8243480 | biostudies-literature
Project description:Profiling analysis aims to evaluate health care providers by modeling each provider's performance with respect to a patient outcome, such as unplanned hospital readmission. High-dimensional regression models are used in profiling to risk-adjust for patient case-mix covariates. Case-mix covariates typically ascertained from administrative databases are inherently error-prone. We examine the impact of case-mix measurement error (ME) on profiling models. The results show that even though the models' coefficient estimates are biased, this does not affect the estimation of standardized readmission ratio (SRR). However, ME leads to increased variation in SRR estimates and degrades the ability to identify under-performing providers.
| S-EPMC7731965 | biostudies-literature
Project description:To develop a model for adjusting patients' reports of behavioral health care experiences on the Experience of Care and Health Outcomes (ECHO) survey to allow for fair comparisons across health plans.Survey responses from 4,068 individuals enrolled in 21 managed behavioral health plans who received behavioral health care within the previous year (response rate = 48 percent).Potential case-mix adjustors were evaluated by combining information about their predictive power and the amount of within- and between-plan variability. Changes in plan scores and rankings due to case-mix adjustment were quantified.The final case-mix adjustment model included self-reported mental health status, self-reported general health status, alcohol/drug treatment, age, education, and race/ethnicity. The impact of adjustment on plan report scores was modest, but large enough to change some plan rankings.Adjusting plan report scores on the ECHO survey for differences in patient characteristics had modest effects, but still may be important to maintain the credibility of patient reports as a quality metric. Differences between those with self-reported fair/poor health compared with those in excellent/very good health varied by plan, suggesting quality differences associated with health status and underscoring the importance of collecting quality information.
| S-EPMC2613989 | biostudies-literature
Project description:BACKGROUND:In stroke studies, ordinal logistic regression (OLR) is often used to analyze outcome on the modified Rankin Scale (mRS), whereas the non-parametric Mann-Whitney measure of superiority (MWS) has also been suggested. It is unclear how these perform comparatively when confounding adjustment is warranted. AIMS:Our aim is to quantify the performance of OLR and MWS in different confounding variable settings. METHODS:We set up a simulation study with three different scenarios; (1) dichotomous confounding variables, (2) continuous confounding variables, and (3) confounding variable settings mimicking a study on functional outcome after stroke. We compared adjusted ordinal logistic regression (aOLR) and stratified Mann-Whitney measure of superiority (sMWS), and also used propensity scores to stratify the MWS (psMWS). For comparability, OLR estimates were transformed to a MWS. We report bias, the percentage of runs that produced a point estimate deviating by more than 0.05 points (point estimate variation), and the coverage probability. RESULTS:In scenario 1, there was no bias in both sMWS and aOLR, with similar point estimate variation and coverage probabilities. In scenario 2, sMWS resulted in more bias (0.04 versus 0.00), and higher point estimate variation (41.6% versus 3.3%), whereas coverage probabilities were similar. In scenario 3, there was no bias in both methods, point estimate variation was higher in the sMWS (6.7%) versus aOLR (1.1%), and coverage probabilities were 0.98 (sMWS) versus 0.95 (aOLR). With psMWS, bias remained 0.00, with less point estimate variation (1.5%) and a coverage probability of 0.95. CONCLUSIONS:The bias of both adjustment methods was similar in our stroke simulation scenario, and the higher point estimate variation in the MWS improved with propensity score based stratification. The stratified MWS is a valid alternative for adjusted OLR only when the ratio of number of strata versus number of observations is relatively low, but propensity score based stratification extends the application range of the MWS.
| S-EPMC7162480 | biostudies-literature
Project description:BackgroundCase mix adjustment is a pre-requisite for valid measurement of healthcare performance and socioeconomic status (SES) is important to account for. Lack of information on individual-level SES has led to investigations into using a proxy for SES based on patient area of residence. The objective of this study was to use neighbourhood SES for case mix adjustment of performance indicators in total hip replacement (THR) in Sweden, and to compare with use of individual SES.MethodsData from patient administrative systems and the Swedish Hip Arthroplasty Register were extracted for all patients undergoing THR in four Swedish regions. For each subject, individual data and neighbourhood data on country of birth, educational level, and income were provided by Statistics Sweden. Three variables were selected for analysis of performance; EQ-5D, hip pain and length of stay (LoS). In addition to socioeconomic information, several important clinical characteristics were used as case mix factors. Regression analysis was used to study each variable's impact on the three outcome variables and model fit was evaluated using mean squared error.ResultsA total of 27,121 patients operated between 2010 and 2016 were included in the study. Both educational level and income were higher when based on neighbourhood information than individual information, while proportion born in Sweden was similar. Higher SES was generally found to be associated with better outcomes and lower LoS, albeit with certain differences between the different measures of SES. The predictive ability of the models was increased when adding information on SES to the clinical characteristics. The increase in predictive ability was higher for individual SES compared to neighbourhood SES. When analysing performance for the two providers with most diverging case mix in terms of SES, the inclusion of SES altered the relative performance using individual as well as neighbourhood SES.ConclusionsIncorporating SES improves case mix adjustment marginally compared to using only clinical information. In this patient group, geographically derived SES was found to improve case mix adjustment compared to only clinical information but not to the same extent as actual individual-level SES.
| S-EPMC7353710 | biostudies-literature
Project description:BackgroundTreatment switching in randomised controlled trials (RCTs) is a problem for health technology assessment when substantial proportions of patients switch onto effective treatments that would not be available in standard clinical practice. Often statistical methods are used to adjust for switching: these can be applied in different ways, and performance has been assessed in simulation studies, but not in real-world case studies. We assessed the performance of adjustment methods described in National Institute for Health and Care Excellence Decision Support Unit Technical Support Document 16, applying them to an RCT comparing panitumumab to best supportive care (BSC) in colorectal cancer, in which 76% of patients randomised to BSC switched onto panitumumab. The RCT resulted in intention-to-treat hazard ratios (HR) for overall survival (OS) of 1.00 (95% confidence interval [CI] 0.82-1.22) for all patients, and 0.99 (95% CI 0.75-1.29) for patients with wild-type KRAS (Kirsten rat sarcoma virus).MethodsWe tested several applications of inverse probability of censoring weights (IPCW), rank preserving structural failure time models (RPSFTM) and simple and complex two-stage estimation (TSE) to estimate treatment effects that would have been observed if BSC patients had not switched onto panitumumab. To assess the performance of these analyses we ascertained the true effectiveness of panitumumab based on: (i) subsequent RCTs of panitumumab that disallowed treatment switching; (ii) studies of cetuximab that disallowed treatment switching, (iii) analyses demonstrating that only patients with wild-type KRAS benefit from panitumumab. These sources suggest the true OS HR for panitumumab is 0.76-0.77 (95% CI 0.60-0.98) for all patients, and 0.55-0.73 (95% CI 0.41-0.93) for patients with wild-type KRAS.ResultsSome applications of IPCW and TSE provided treatment effect estimates that closely matched the point-estimates and CIs of the expected truths. However, other applications produced estimates towards the boundaries of the expected truths, with some TSE applications producing estimates that lay outside the expected true confidence intervals. The RPSFTM performed relatively poorly, with all applications providing treatment effect estimates close to 1, often with extremely wide confidence intervals.ConclusionsAdjustment analyses may provide unreliable results. How each method is applied must be scrutinised to assess reliability.
| S-EPMC10802004 | biostudies-literature
Project description:Compaction index is one of the most important technological parameters during asphalt pavement construction which may be negatively affected by wrong asphalt paving machine set, weather conditions, or the mix temperature. Presented laboratory study analyzes the asphalt mix properties in case of inappropriate compaction. The reference mix was designed for AC 11 S wearing layer (asphalt concrete for wearing layer with maximum grading of 11 mm). Asphalt mix samples used in the tests were prepared using Marshall device with the compaction energy of 2 × 20, 2 × 35, 2 × 50, and 2 × 75 blows as well as in a roller compactor where the slabs were compacted to various heights: 69.3 mm (+10% of nominal height), 66.2 mm (+5%), 63 mm (nominal), and 59.9 mm (-5%) which resulted in different compaction indexes. Afterwards the samples were cored from the slabs. Both Marshall samples and cores were tested for air void content, stiffness modulus in three temperatures, indirect tensile strength, and resistance to water and frost indicated by ITSR value. It was found that either insufficient or excessive level of compaction can cause negative effect on the road surface performance.
| S-EPMC7321419 | biostudies-literature
Project description:BackgroundCaesarean delivery (CD) rates are commonly used as an indicator of quality in obstetric care and risk adjustment evaluation is recommended to assess inter-institutional variations. The aim of this study was to evaluate whether the Ten Group classification system (TGCS) can be used in case-mix adjustment.MethodsStandardized data on 15,255 deliveries from 11 different regional centers were prospectively collected. Crude Risk Ratios of CDs were calculated for each center. Two multiple logistic regression models were herein considered by using: Model 1- maternal (age, Body Mass Index), obstetric variables (gestational age, fetal presentation, single or multiple, previous scar, parity, neonatal birth weight) and presence of risk factors; Model 2- TGCS either with or without maternal characteristics and presence of risk factors. Receiver Operating Characteristic (ROC) curves of the multivariate logistic regression analyses were used to assess the diagnostic accuracy of each model. The null hypothesis that Areas under ROC Curve (AUC) were not different from each other was verified with a Chi Square test and post hoc pairwise comparisons by using a Bonferroni correction.ResultsCrude evaluation of CD rates showed all centers had significantly higher Risk Ratios than the referent. Both multiple logistic regression models reduced these variations. However the two methods ranked institutions differently: model 1 and model 2 (adjusted for TGCS) identified respectively nine and eight centers with significantly higher CD rates than the referent with slightly different AUCs (0.8758 and 0.8929 respectively). In the adjusted model for TGCS and maternal characteristics/presence of risk factors, three centers had CD rates similar to the referent with the best AUC (0.9024).ConclusionsThe TGCS might be considered as a reliable variable to adjust CD rates. The addition of maternal characteristics and risk factors to TGCS substantially increase the predictive discrimination of the risk adjusted model.
| S-EPMC3674002 | biostudies-literature