Dataset Information

Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models.

ABSTRACT: When comparing performances of two risk prediction models, several metrics exist to quantify prognostic improvement, including the change in the area under the Receiver Operating Characteristic curve, the Integrated Discrimination Improvement, the Net Reclassification Index at event rate, the change in Standardized Net Benefit, the change in Brier score, and the change in scaled Brier score. We explore the behavior and interrelationships between these metrics under multivariate normality in nested and nonnested model comparisons. We demonstrate that, within the framework of linear discriminant analysis, all six statistics are functions of squared Mahalanobis distance, a robust metric that properly measures discrimination by quantifying the separation between the risk scores of events and nonevents. These relationships are important for overall interpretability and clinical usefulness. Through simulation, we demonstrate that the performance of the theoretical estimators under normality is comparable or superior to empirical estimation methods typically used by investigators. In particular, the theoretical estimators for the Net Reclassification Index and the change in Standardized Net Benefit exhibit less variability in their estimates as compared to their empirically estimated counterparts. Finally, we explore how these metrics behave with potentially nonnormal data by applying these methods in a practical example based on the sex-specific cardiovascular disease risk models from the Framingham Heart Study. Our findings aim to give greater insight into the behavior of these measures and the connections existing among them and to provide additional estimation methods with less variability for the Net Reclassification Index and the change in Standardized Net Benefit.

SUBMITTER: Enserro DM

PROVIDER: S-EPMC6827341 | biostudies-literature | 2019 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models.

Enserro Danielle M DM Demler Olga V OV Pencina Michael J MJ D'Agostino Ralph B RB

Statistics in medicine 20190618 20

When comparing performances of two risk prediction models, several metrics exist to quantify prognostic improvement, including the change in the area under the Receiver Operating Characteristic curve, the Integrated Discrimination Improvement, the Net Reclassification Index at event rate, the change in Standardized Net Benefit, the change in Brier score, and the change in scaled Brier score. We explore the behavior and interrelationships between these metrics under multivariate normality in nest ...[more]

PMID: 31211443

Similar Datasets

Project description:The detection of causal effects among simultaneous observations provides knowledge about the underlying network, and is a topic of interests in many scientific areas. Over the years different causality measures have been developed, each with their own advantages and disadvantages. However, an extensive evaluation study is missing. In this work we consider some of the best-known causality measures i.e., cross-correlation, (conditional) Granger causality index (CGCI), partial directed coherence (PDC), directed transfer function (DTF), and partial mutual information on mixed embedding (PMIME). To correct for noise-related spurious connections, each measure (except PMIME) is tested for statistical significance based on surrogate data. The performance of the causality metrics is evaluated on a set of simulation models with distinct characteristics, to assess how well they work in- as well as outside of their "comfort zone." PDC and DTF perform best on systems with frequency-specific connections, while PMIME is the only one able to detect non-linear interactions. The varying performance depending on the system characteristics warrants the use of multiple measures and comparing their results to avoid errors. Furthermore, lags between coupled variables are inherent to real-world systems and could hold essential information on the network dynamics. They are however often not taken into account and we lack proper tools to estimate them. We propose three new methods for lag estimation in multivariate time series, based on autoregressive modelling and information theory. One of the autoregressive methods and the one based on information theory were able to reliably identify the correct lag value in different simulated systems. However, only the latter was able to maintain its performance in the case of non-linear interactions. As a clinical application, the same methods are also applied on an intracranial recording of an epileptic seizure. The combined knowledge from the causality measures and insights from the simulations, on how these measures perform under different circumstances and when to use which one, allow us to recreate a plausible network of the seizure propagation that supports previous observations of desynchronisation and synchronisation during seizure progression. The lag estimation results show absence of a relationship between connectivity strength and estimated lag values, which contradicts the line of thinking in connectivity shaped by the neuron doctrine.

Project description:ObjectivesPatients with severe spontaneous intracranial haemorrhages, managed in intensive care units, face ethical issues regarding the difficulty of anticipating their recovery. Prognostic tools help clinicians in counselling patients and relatives and guide therapeutic decisions. We aimed to methodologically assess prognostic tools for functional outcomes in severe spontaneous intracranial haemorrhages.Data sourcesFollowing Preferred Reporting Items for Systematic Reviews and Meta-Analyses recommendations, we conducted a systematic review querying Medline, Embase, Web of Science, and the Cochrane in January 2020.Study selectionWe included development or validation of multivariate prognostic models for severe intracerebral or subarachnoid haemorrhage.Data extractionWe evaluated the articles following the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies and Transparent Reporting of multivariable prediction model for Individual Prognosis Or Diagnosis statements to assess the tools' methodological reporting.ResultsOf the 6149 references retrieved, we identified 85 articles eligible. We discarded 43 articles due to the absence of prognostic performance or predictor selection. Among the 42 articles included, 22 did not validate models, 6 developed and validated models and 14 only externally validated models. When adding 11 articles comparing developed models to existing ones, 25 articles externally validated models. We identified methodological pitfalls, notably the lack of adequate validations or insufficient performance levels. We finally retained three scores predicting mortality and unfavourable outcomes: the IntraCerebral Haemorrhages (ICH) score and the max-ICH score for intracerebral haemorrhages, the SubArachnoid Haemorrhage International Trialists score for subarachnoid haemorrhages.ConclusionsAlthough prognostic studies on intracranial haemorrhages abound in the literature, they lack methodological robustness or show incomplete reporting. Rather than developing new scores, future authors should focus on externally validating and updating existing scores with large and recent cohorts.

Project description:Background: Voxel-based morphometry studies have repeatedly highlighted the presence of distributed gray matter changes in schizophrenia, but to date, it is not clear if clinically useful prognostic information can be gleaned from structural imaging. The suspected association between gray matter volume (GMV) and duration of psychotic illness, antipsychotic exposure, and symptom severity also limits the prognostic utility of morphometry. We address the question of whether morphometric information from patients with drug-naive first-episode psychosis can predict the linear trajectory of symptoms following early antipsychotic intervention using a longitudinal design. Method: Sixty-two first-episode, drug-naive patients with schizophrenia underwent brain magnetic resonance imaging scans at baseline, treated with antipsychotics, and rescanned after 1-year follow-up. Positive and Negative Syndrome Scale (PANSS) was used to assess their clinical manifestations. A multivariate approach to detect covariance-based network-like spatial components [Source Based Morphometry (SBM)] was performed to analyze the GMV. Paired t tests were used to study changes in the loading coefficients of GMV in the spatial components between two time points. The reduction in PANSS scores between the baseline (T0) and 1-year follow-up (T1) expressed as a ratio of the baseline scores (reduction ratio) was computed for positive, negative, and disorganization symptoms. Separate multiple regression analyses were conducted to predict the longitudinal change in symptoms (treatment response) using the loading coefficients of spatial components that differed between T0 and T1 with age, gender, duration of illness, and antipsychotic dose as covariates. We also tested the putative "toxicity" effects of baseline symptom severity on the GMV at 1 year using multiple regression analysis. Results: Of the 30 spatial components of gray matter extracted using SBM, loading coefficients of anterior cingulate cortex (ACC), insula and inferior frontal gyrus (IFG), superior temporal gyrus (STG), middle temporal gyrus (MTG), precuenus, and dorsolateral prefrontal cortex (DLPFC) reduced with time in patients. Specifically, the lower volume of insula and IFG at baseline predicted a lack of improvement in positive and disorganization symptoms. None of the symptom severity scores (positive, negative, or disorganization) at baseline independently predicted the reduced GMV at 1 year. Conclusions: The baseline deficit in a covariance-based network-like spatial component comprising of insula and IFG is predictive of the clinical course of schizophrenia. We do not find any evidence to support the notion of symptoms per se being neurotoxic to gray matter tissue. If judiciously combined with other available predictors of clinical outcome, multivariate morphometric information can improve our ability to predict prognosis in schizophrenia.

Project description:Developing materials for use in photovoltaic (PV) systems requires knowledge of their performance over the warranted lifetime of the PV system. Poly(ethylene-terephthalate) (PET) is a critical component of PV module backsheets due to its dielectric properties and low cost. However, PET is susceptible to environmental stressors and degrades over time. Changes in the physical properties of nine PET grades were modeled after outdoor and accelerated weathering exposures to characterize the degradation process of PET and assess the influence of stabilizing additives and weathering factors. Multivariate multiple regression (MMR) models were developed to quantify changes in color, gloss, and haze of the materials. Natural splines were used to capture the non-linear relationship between predictors and responses. Model performance was evaluated via adjusted-R2 and root mean squared error values from leave-one-out cross validation analysis. All models described over 85% of the variation in the data with low relative error. Model coefficients were used to assess the influence of weathering stressors and material additives on the property changes of films. Photodose was found to be the primary degradation stressor and moisture was found to increase the degradation rate of PET. Direct moisture contact was found to impose more stress on the material than airbone moisture (humidity). Increasing the concentration of TiO2 was found to generally decrease the degradation rate of PET and mitigate hydrolytic degradation. MMR models were compared to physics-based models and agreement was found between the two modeling approaches. Cross-correlation of accelerated exposures to outdoor exposures was achieved via determination of cross-correlation scale factors. Cross-correlation revealed that direct moisture contact is a key factor for reliable accelerated weathering testing and provided a quantitative method to determine when accelerated exposure results can be made more aggressive to better approximate outdoor exposure conditions.

Dataset Information

Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models.

Publications

Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets