Dataset Information

A distributional multivariate approach for assessing performance of climate-hydrology models.

ABSTRACT: One of the ultimate goals of climate studies is to provide projections of future scenarios: for this purpose, sophisticated models are conceived, involving lots of parameters calibrated via observed data. The outputs of such models are used to investigate the impacts on related phenomena such as floods, droughts, etc. To evaluate the performance of such models, statistics like moments/quantiles are used, and comparisons with historical data are carried out. However, this may not be enough: correct estimates of some moments/quantiles do not imply that the probability distributions of observed and simulated data match. In this work, a distributional multivariate approach is outlined, also accounting for the fact that climate variables are often dependent. Suitable statistical tests are described, providing a non-parametric assessment exploiting the Copula Theory. These procedures allow to understand (i) whether the models are able to reproduce the distributional features of the observations, and (ii) how the models perform (e.g., in terms of future climate projections and changes). The proposed methodological approach is appropriate also in contexts different from climate studies, to evaluate the performance of any model of interest: methods to check a model per se are sketched out, investigating whether its outcomes are (statistically) consistent.

SUBMITTER: Vezzoli R

PROVIDER: S-EPMC5608904 | biostudies-other | 2017 Sep

REPOSITORIES: biostudies-other

ACCESS DATA

Publications

A distributional multivariate approach for assessing performance of climate-hydrology models.

Vezzoli Renata R Salvadori Gianfausto G De Michele Carlo C

Scientific reports 20170921 1

One of the ultimate goals of climate studies is to provide projections of future scenarios: for this purpose, sophisticated models are conceived, involving lots of parameters calibrated via observed data. The outputs of such models are used to investigate the impacts on related phenomena such as floods, droughts, etc. To evaluate the performance of such models, statistics like moments/quantiles are used, and comparisons with historical data are carried out. However, this may not be enough: corre ...[more]

PMID: 28935876

Similar Datasets

Project description:BackgroundFast, memory-efficient, and reliable algorithms for estimating genomic estimated breeding values (GEBV) for multiple traits and environments are needed to make timely decisions in breeding. Multivariate genomic prediction exploits genetic correlations between traits and environments to increase accuracy of GEBV compared to univariate methods. These genetic correlations are estimated simultaneously with GEBV, because they are specific to year, environment, and management. However, estimating genetic parameters is computationally demanding with restricted maximum likelihood (REML) and Bayesian samplers, and canonical transformations or orthogonalizations cannot be used for unbalanced experimental designs.MethodsWe propose a multivariate randomized Gauss-Seidel algorithm for simultaneous estimation of model effects and genetic parameters. Two previously proposed methods for estimating genetic parameters were combined with a Gauss-Seidel (GS) solver, and were called Tilde-Hat-GS (THGS) and Pseudo-Expectation-GS (PEGS). Balanced and unbalanced experimental designs were simulated to compare runtime, bias and accuracy of GEBV, and bias and standard errors of estimates of heritabilities and genetic correlations of THGS, PEGS, and REML. Models with 10 to 400 response variables, 1279 to 42,034 genetic markers, and 5990 to 1.85 million observations were fitted.ResultsRuntime of PEGS and THGS was a fraction of REML. Accuracies of GEBV were slightly lower than those from REML, but higher than those from the univariate approach, hence THGS and PEGS exploited genetic correlations. For 500 to 600 observations per response variable, biases of estimates of genetic parameters of THGS and PEGS were small, but standard errors of estimates of genetic correlations were higher than for REML. Bias and standard errors decreased as sample size increased. For balanced designs, GEBV and estimates of genetic correlations from THGS were unbiased when only an intercept and eigenvectors of genotype scores were fitted.ConclusionsTHGS and PEGS are fast and memory-efficient algorithms for multivariate genomic prediction for balanced and unbalanced experimental designs. They are scalable for increasing numbers of environments and genetic markers. Accuracy of GEBV was comparable to REML. Estimates of genetic parameters had little bias, but their standard errors were larger than for REML. More studies are needed to evaluate the proposed methods for datasets that contain selection.

Project description:Multivariate decoding models are increasingly being applied to functional magnetic imaging (fMRI) data to interpret the distributed neural activity in the human brain. These models are typically formulated to optimize an objective function that maximizes decoding accuracy. For decoding models trained on full-brain data, this can result in multiple models that yield the same classification accuracy, though some may be more reproducible than others--i.e. small changes to the training set may result in very different voxels being selected. This issue of reproducibility can be partially controlled by regularizing the decoding model. Regularization, along with the cross-validation used to estimate decoding accuracy, typically requires retraining many (often on the order of thousands) of related decoding models. In this paper we describe an approach that uses a combination of bootstrapping and permutation testing to construct both a measure of cross-validated prediction accuracy and model reproducibility of the learned brain maps. This requires re-training our classification method on many re-sampled versions of the fMRI data. Given the size of fMRI datasets, this is normally a time-consuming process. Our approach leverages an algorithm called fast simultaneous training of generalized linear models (FaSTGLZ) to create a family of classifiers in the space of accuracy vs. reproducibility. The convex hull of this family of classifiers can be used to identify a subset of Pareto optimal classifiers, with a single-optimal classifier selectable based on the relative cost of accuracy vs. reproducibility. We demonstrate our approach using full-brain analysis of elastic-net classifiers trained to discriminate stimulus type in an auditory and visual oddball event-related fMRI design. Our approach and results argue for a computational approach to fMRI decoding models in which the value of the interpretation of the decoding model ultimately depends upon optimizing a joint space of accuracy and reproducibility.

Project description:IntroductionExternal validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.MethodsWe compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.ResultsThe permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.ConclusionThe recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.

Dataset Information

A distributional multivariate approach for assessing performance of climate-hydrology models.

Publications

A distributional multivariate approach for assessing performance of climate-hydrology models.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets