Assessing discrimination of risk prediction rules in a clustered data setting.
Ontology highlight
ABSTRACT:
SUBMITTER: Rosner B
PROVIDER: S-EPMC3622772 | biostudies-literature | 2013 Apr
REPOSITORIES: biostudies-literature
Publications
Assessing discrimination of risk prediction rules in a clustered data setting.
Rosner Bernard B Qiu Weiliang W Lee Mei-Ling T ML
Lifetime data analysis 20121222 2
The AUC (area under ROC curve) is a commonly used metric to assess discrimination of risk prediction rules; however, standard errors of AUC are usually based on the Mann-Whitney U test that assumes independence of sampling units. For ophthalmologic applications, it is desirable to assess risk prediction rules based on eye-specific outcome variables which are generally highly, but not perfectly correlated in fellow eyes [e.g. progression of individual eyes to age-related macular degeneration (AMD ...[more]
Similar Datasets
Project description:Individual participant time-to-event data from multiple prospective epidemiologic studies enable detailed investigation into the predictive ability of risk models. Here we address the challenges in appropriately combining such information across studies. Methods are exemplified by analyses of log C-reactive protein and conventional risk factors for coronary heart disease in the Emerging Risk Factors Collaboration, a collation of individual data from multiple prospective studies with an average follow-up duration of 9.8 years (dates varied). We derive risk prediction models using Cox proportional hazards regression analysis stratified by study and obtain estimates of risk discrimination, Harrell's concordance index, and Royston's discrimination measure within each study; we then combine the estimates across studies using a weighted meta-analysis. Various weighting approaches are compared and lead us to recommend using the number of events in each study. We also discuss the calculation of measures of reclassification for multiple studies. We further show that comparison of differences in predictive ability across subgroups should be based only on within-study information and that combining measures of risk discrimination from case-control studies and prospective studies is problematic. The concordance index and discrimination measure gave qualitatively similar results throughout. While the concordance index was very heterogeneous between studies, principally because of differing age ranges, the increments in the concordance index from adding log C-reactive protein to conventional risk factors were more homogeneous.
| S-EPMC3927974 | biostudies-literature
Project description:BackgroundClinical prediction models often fail to generalize in the context of clustered data, because most models fail to account for heterogeneity in outcome values and covariate effects across clusters. Furthermore, standard approaches for modeling clustered data, including generalized linear mixed-effects models, would not be expected to provide accurate predictions in novel clusters, because such predictions are typically based on the hypothetical mean cluster. We hypothesized that dynamic mixed-effects models, which incorporate data from previous predictions to refine the model for future predictions, would allow for cluster-specific predictions in novel clusters as the model is updated over time, thus improving overall model generalizability.ResultsWe quantified the potential gains in prediction accuracy from using a dynamic modeling strategy in a simulation study. Furthermore, because clinical prediction models in the context of clustered data often involve outcomes that are dependent on patient volume, we examined whether using dynamic mixed-effects models would be robust to misspecification of the volume-outcome relationship. Our results indicated that dynamic mixed-effects models led to substantial improvements in prediction accuracy in clustered populations over a broad range of conditions, and were uniformly superior to static models. In addition, dynamic mixed-effects models were particularly robust to misspecification of the volume-outcome relationship and to variation in the frequency of model updating. The extent of the improvement in prediction accuracy that was observed with dynamic mixed-effects models depended on the relative impact of fixed and random effects on the outcome as well as the degree of misspecification of model fixed effects.ConclusionsDynamic mixed-effects models led to substantial improvements in prediction model accuracy across a broad range of simulated conditions. Therefore, dynamic mixed-effects models could be a useful alternative to standard static models for improving the generalizability of clinical prediction models in the setting of clustered data, and, thus, well worth the logistical challenges that may accompany their implementation in practice.
| S-EPMC4728760 | biostudies-literature
Project description:Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor-outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal-external cross-validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold-out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta-analysis of calibration and discrimination performance in each hold-out cluster shows that trade-offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.
| S-EPMC8252590 | biostudies-literature
Project description:BackgroundWhen study data are clustered, standard regression analysis is considered inappropriate and analytical techniques for clustered data need to be used. For prediction research in which the interest of predictor effects is on the patient level, random effect regression models are probably preferred over standard regression analysis. It is well known that the random effect parameter estimates and the standard logistic regression parameter estimates are different. Here, we compared random effect and standard logistic regression models for their ability to provide accurate predictions.MethodsUsing an empirical study on 1642 surgical patients at risk of postoperative nausea and vomiting, who were treated by one of 19 anesthesiologists (clusters), we developed prognostic models either with standard or random intercept logistic regression. External validity of these models was assessed in new patients from other anesthesiologists. We supported our results with simulation studies using intra-class correlation coefficients (ICC) of 5%, 15%, or 30%. Standard performance measures and measures adapted for the clustered data structure were estimated.ResultsThe model developed with random effect analysis showed better discrimination than the standard approach, if the cluster effects were used for risk prediction (standard c-index of 0.69 versus 0.66). In the external validation set, both models showed similar discrimination (standard c-index 0.68 versus 0.67). The simulation study confirmed these results. For datasets with a high ICC (≥15%), model calibration was only adequate in external subjects, if the used performance measure assumed the same data structure as the model development method: standard calibration measures showed good calibration for the standard developed model, calibration measures adapting the clustered data structure showed good calibration for the prediction model with random intercept.ConclusionThe models with random intercept discriminate better than the standard model only if the cluster effect is used for predictions. The prediction model with random intercept had good calibration within clusters.
| S-EPMC3658967 | biostudies-literature
Project description:Discriminating groups were introduced by G. Baumslag, A. Myasnikov, and V. Remeslennikov as an outgrowth of their theory of algebraic geometry over groups. Algebraic geometry over groups became the main method of attack on the solution of the celebrated Tarski conjectures. In this paper we explore the notion of discrimination in a general universal algebra context. As an application we provide a different proof of a theorem of Malcev on axiomatic classes of Ω-algebras.
| S-EPMC4485932 | biostudies-other
Project description:Protein structure and function can be severely altered by even a single amino acid mutation. Predictions of mutational effects using extensive artificial intelligence (AI)-based models, although accurate, remain as enigmatic as the experimental observations in terms of improving intuitions about the contributions of various factors. Inspired by Lipinski's rules for drug-likeness, we devise simple thresholding criteria on five different descriptors such as conservation, which have so far been limited to qualitative interpretations such as high conservation implies high mutational effect. We analyze systematic deep mutational scanning data of all possible single amino acid substitutions on seven proteins (25153 mutations) to first define these thresholds and then to evaluate the scope and limits of the predictions. At this stage, the approach allows us to comment easily and with a low error rate on the subset of mutations classified as neutral or deleterious by all of the descriptors. We hope that complementary to the accurate AI predictions, these thresholding rules or their subsequent modifications will serve the purpose of codifying the knowledge about the effects of mutations.
| S-EPMC7689672 | biostudies-literature
Project description:The risk difference scale is often of primary interest when evaluating public health impacts of interventions on binary outcomes. However, few investigators report findings in terms of additive interaction, probably because the models typically used for binary outcomes implicitly measure interaction on the multiplicative scale. One measure with which to assess additive interaction from multiplicative models is the relative excess risk due to interaction (RERI). The RERI measure has been applied in many contexts, but one limitation of previous approaches is that clustering in data has rarely been considered. We evaluated the RERI metric for the setting of clustered data using both population-averaged and cluster-conditional models. In simulation studies, we found that estimation and inference for the RERI using population-averaged models was straightforward. However, frequentist implementations of cluster-conditional models including random intercepts often failed to converge or produced degenerate variance estimates. We developed a Bayesian implementation of log binomial random-intercept models, which represents an attractive alternative for estimating the RERI in cluster-conditional models. We applied the methods to an observational study of adverse birth outcomes in mothers with human immunodeficiency virus, in which mothers were clustered within clinical research sites.
| S-EPMC6211249 | biostudies-literature
Project description:BackgroundRandom effects modelling is routinely used in clustered data, but for prediction models, random effects are commonly substituted with the mean zero after model development. In this study, we proposed a novel approach of including prior knowledge through the random effects distribution and investigated to what extent this could improve the predictive performance.MethodsData were simulated on the basis of a random effects logistic regression model. Five prediction models were specified: a frequentist model that set the random effects to zero for all new clusters, a Bayesian model with weakly informative priors for the random effects of new clusters, Bayesian models with expert opinion incorporated into low informative, medium informative and highly informative priors for the random effects. Expert opinion at the cluster level was elicited in the form of a truncated area of the random effects distribution. The predictive performance of the five models was assessed. In addition, impact of suboptimal expert opinion that deviated from the true quantity as well as including expert opinion by means of a categorical variable in the frequentist approach were explored. The five models were further investigated in various sensitivity analyses.ResultsThe Bayesian prediction model using weakly informative priors for the random effects showed similar results to the frequentist model. Bayesian prediction models using expert opinion as informative priors showed smaller Brier scores, better overall discrimination and calibration, as well as better within cluster calibration. Results also indicated that incorporation of more precise expert opinion led to better predictions. Predictive performance from the frequentist models with expert opinion incorporated as categorical variable showed similar patterns as the Bayesian models with informative priors. When suboptimal expert opinion was used as prior information, results indicated that prediction still improved in certain settings.ConclusionsThe prediction models that incorporated cluster level information showed better performance than the models that did not. The Bayesian prediction models we proposed, with cluster specific expert opinion incorporated as priors for the random effects showed better predictive ability in new data, compared to the frequentist method that replaced random effects with zero after model development.
| S-EPMC6080562 | biostudies-literature
Project description:Despite development of new technologies for caries control, tooth decay in primary teeth remains a major global health problem. Caries risk assessment (CRA) models for toddlers and preschoolers are rare. Among them, almost all models use dental factors (e.g., past caries experience) to predict future caries risk, with limited clinical/community applicability owing to relatively uncommon dental visits compared to frequent medical visits during the first year of life. The objective of this study was to construct and evaluate risk prediction models using information easily accessible to medical practitioners to forecast caries at 2 and 3 y of age. Data were obtained from the Growing Up in Singapore Towards Healthy Outcomes (GUSTO) mother-offspring cohort. Caries was diagnosed using modified International Caries Detection and Assessment System criteria. Risk prediction models were constructed using multivariable logistic regression coupled with receiver operating characteristic analyses. Imputation was performed using multiple imputation by chained equations to assess effect of missing data. Caries rates at ages 2 y (n = 535) and 3 y (n = 721) were 17.8% and 42.9%, respectively. Risk prediction models predicting overall caries risk at 2 and 3 y demonstrated area under the curve (AUC) (95% confidence interval) of 0.81 (0.75-0.87) and 0.79 (0.74-0.84), respectively, while those predicting moderate to extensive lesions showed 0.91 (0.85-0.97) and 0.79 (0.73-0.85), respectively. Postimputation results showed reduced AUC of 0.75 (0.74-0.81) and 0.71 (0.67-0.75) at years 2 and 3, respectively, for overall caries risk, while AUC was 0.84 (0.76-0.92) and 0.75 (0.70-0.80), respectively, for moderate to extensive caries. Addition of anterior caries significantly increased AUC in all year 3 models with or without imputation (all P < 0.05). Significant predictors/protectors were identified, including ethnicity, prenatal tobacco smoke exposure, history of allergies before 12 mo, history of chronic maternal illness, maternal brushing frequency, childbearing age, and so on. Integrating oral-general health care using medical CRA models may be promising in screening caries-susceptible infants/toddlers, especially when medical professionals are trained to "lift the lip" to identify anterior caries lesions.
| S-EPMC7343522 | biostudies-literature
Project description:Predicting individual large earthquakes (EQs)' locations, magnitudes, and timing remains unreachable. The author's prior study shows that individual large EQs have unique signatures obtained from multi-layered data transformations. Via spatio-temporal convolutions, decades-long EQ catalog data are transformed into pseudo-physics quantities (e.g., energy, power, vorticity, and Laplacian), which turn into surface-like information via Gauss curvatures. Using these new features, a rule-learning machine learning approach unravels promising prediction rules. This paper suggests further data transformation via Fourier transformation (FT). Results show that FT-based new feature can help sharpen the prediction rules. Feasibility tests of large EQs ([Formula: see text] 6.5) over the past 40 years in the western U.S. show promise, shedding light on data-driven prediction of individual large EQs. The handshake among ML methods, Fourier, and Gauss may help answer the long-standing enigma of seismogenesis.
| S-EPMC10520116 | biostudies-literature