Is precipitation a good metric for model performance?
Ontology highlight
ABSTRACT:
SUBMITTER: Tapiador FJ
PROVIDER: S-EPMC6951255 | biostudies-literature | 2019 Mar
REPOSITORIES: biostudies-literature
Publications
Is precipitation a good metric for model performance?
Tapiador Francisco J FJ Roca Rémy R Del Genio Anthony A Dewitte Boris B Petersen Walt W Zhang Fuqing F
Bulletin of the American Meteorological Society 20190301 2
Precipitation has often been used to gauge the performances of numerical weather and climate models, sometimes together with other variables such as temperature, humidity, geopotential, and clouds. Precipitation, however, is singular in that it can present a high spatial variability and probably the sharpest gradients amongst all meteorological fields. Moreover, its quantitative measurement is plagued with difficulties and there are even notable differences among different reference datasets. Se ...[more]
Similar Datasets
Project description:The surge of interest in personalized and precision medicine during recent years has increased the application of ordinal classification problems in biomedical science. Currently, accuracy, Kendall's ?b , and average mean absolute error are three commonly used metrics for evaluating the effectiveness of an ordinal classifier. Although there are benefits to each, no single metric considers the benefits of predictive accuracy with the tradeoffs of misclassification cost. In addition, decision analysis that considers pairwise analysis of the metrics is not trivial due to inconsistent findings. A new cost-sensitive metric is proposed to find the optimal tradeoff between the two most critical performance measures of a classification task - accuracy and cost. The proposed method accounts for an inherent ordinal data structure, total misclassification cost of a classifier, and imbalanced class distribution. The strengths of the new methodology are demonstrated through analyses of three real cancer datasets and four simulation studies. The new cost-sensitive metric proved better performance in its ability to identify the best ordinal classifier for a given analysis. The performance metric devised in this study provides a comprehensive tool for comparative analysis of multiple (and competing) ordinal classifiers. Consideration of the tradeoff between accuracy and misclassification cost in decisions regarding ordinal classification problems is imperative in real-world application. The work presented here is a precursor to the possibility of incorporating the proposed metric into a prediction modeling algorithm for ordinal data as a means of integrating misclassification cost in final model selection.
| S-EPMC5217743 | biostudies-literature
Project description:Objectives:The Emergency Medicine Milestone Project, a framework for assessing competencies, has been used as a method of providing focused resident feedback. However, the emergency medicine milestones do not include specific objective data about resident clinical efficiency and productivity, and studies have shown that milestone-based feedback does not improve resident satisfaction with the feedback process. We examined whether providing performance metric reports to resident physicians improves their satisfaction with the feedback process and their clinical performance. Methods:We conducted a three-phase stepped-wedge randomized pilot study of emergency medicine residents at a single, urban academic site. In phase 1, all residents received traditional feedback; in phase 2, residents were randomized to receive traditional feedback (control group) or traditional feedback with performance metric reports (intervention group); and in phase 3, all residents received monthly performance metric reports and traditional feedback. To assess resident satisfaction with the feedback process, surveys using 6-point Likert scales were administered at each study phase and analyzed using two-sample t-tests. Analysis of variance in repeated measures was performed to compare impact of feedback on resident clinical performance, specifically patient treatment time (PTT) and patient visits per hour. Results:Forty-one residents participated in the trial of which 21 were randomized to the intervention group and 20 in the control group. Ninety percent of residents liked receiving the report and 74% believed that it better prepared them for expectations of becoming an attending physician. Additionally, residents randomized to the intervention group reported higher satisfaction (p = 0.01) with the quality of the feedback compared to residents in the control group. However, receiving performance metric reports, regardless of study phase or postgraduate year status, did not affect clinical performance, specifically PTT (183 minutes vs. 177 minutes, p = 0.34) or patients visits per hour (0.99 vs. 1.04, p = 0.46). Conclusions:While feedback with performance metric reports did not improve resident clinical performance, resident physicians were more satisfied with the feedback process, and a majority of residents expressed liking the reports and felt that it better prepared them to become attending physicians. Residency training programs could consider augmenting feedback with performance metric reports to aide in the transition from resident to attending physician.
| S-EPMC6795364 | biostudies-literature
Project description:BACKGROUND:Interventions to improve performance of global programs in the HIV cascade of care are widespread and increasing the focus of implementation science. At present, however, there is no clear consensus on how to conceptualize their improvement at the program level. The commonly used measures of association, based on ratios of probabilities (or odds), have well-known defects in public health applications. They yield large effect sizes even when the absolute effects, and therefore the public health impact, are small. On the other hand, risk differences create problems because settings with higher baseline values are penalized. We aim to examine ways of quantifying improvement in each health center of a cluster-randomized trial in Uganda to accelerate antiretroviral therapy initiation among HIV-infected adults. METHODS:We formalize the concept of the 'improvement index,' defined as the fraction of gaps closed as a metric of improvement, and suggest that it has unique features and strengths when compared to risk ratios and risk differences. RESULTS:Overall agreement between the different indices was not high, especially among health centers that were among the top 5 or 10. However, all ranking showed broad similarities at the far ends of the spectrum. On scatter plots, there was a positive linear relationship between the metrics, and the Bland Altman (B-A) plots were in agreement. CONCLUSION:The improvement index can be used as an alternative measure of association in implementation science interventions. It can be useful for public health purposes as it demonstrates how much can be covered from the baseline.
| S-EPMC7428843 | biostudies-literature
Project description:Estimating age based on neuroimaging-derived data has become a popular approach to developing markers for brain integrity and health. While a variety of machine-learning algorithms can provide accurate predictions of age based on brain characteristics, there is significant variation in model accuracy reported across studies. We predicted age in two population-based datasets, and assessed the effects of age range, sample size and age-bias correction on the model performance metrics Pearson's correlation coefficient (r), the coefficient of determination (R2 ), Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE). The results showed that these metrics vary considerably depending on cohort age range; r and R2 values are lower when measured in samples with a narrower age range. RMSE and MAE are also lower in samples with a narrower age range due to smaller errors/brain age delta values when predictions are closer to the mean age of the group. Across subsets with different age ranges, performance metrics improve with increasing sample size. Performance metrics further vary depending on prediction variance as well as mean age difference between training and test sets, and age-bias corrected metrics indicate high accuracy-also for models showing poor initial performance. In conclusion, performance metrics used for evaluating age prediction models depend on cohort and study-specific data characteristics, and cannot be directly compared across different studies. Since age-bias corrected metrics generally indicate high accuracy, even for poorly performing models, inspection of uncorrected model results provides important information about underlying model attributes such as prediction variance.
| S-EPMC9188975 | biostudies-literature
Project description:The details surrounding the cross-over from wormlike-specific to universal polymeric behavior has been the subject of debate and confusion even for the simple case of a dilute, unconfined wormlike chain. We have directly computed the polymer size, form factor, free energy and Kirkwood diffusivity for unconfined wormlike chains as a function of molecular weight, focusing on persistence lengths and effective widths that represent single-stranded and double-stranded DNA in a high ionic strength buffer. To do so, we use a chain-growth Monte Carlo algorithm, the Pruned-Enriched Rosenbluth Method (PERM), which allows us to estimate equilibrium and near-equilibrium dynamic properties of wormlike chains over an extremely large range of contour lengths. From our calculations, we find that very large DNA chains (≈ 1,000,000 base pairs depending on the choice of size metric) are required to reach flexible, swollen non-draining coils. Furthermore, our results indicate that the commonly used model polymer λ-DNA (48,500 base pairs) does not exhibit "ideal" scaling, but exists in the middle of the transition to long-chain behavior. We subsequently conclude that typical DNA used in experiments are too short to serve as an accurate model of long-chain, universal polymer behavior.
| S-EPMC3859536 | biostudies-literature
Project description:The field of habitat ecology has been muddled by imprecise terminology regarding what constitutes habitat, and how importance is measured through use, selection, avoidance and other bio-statistical terminology. Added to the confusion is the idea that habitat is scale-specific. Despite these conceptual difficulties, ecologists have made advances in understanding 'how habitats are important to animals', and data from animal-borne global positioning system (GPS) units have the potential to help this clarification. Here, we propose a new conceptual framework to connect habitats with measures of animal performance itself--towards assessing habitat-performance relationship (HPR). Long-term studies will be needed to estimate consequences of habitat selection for animal performance. GPS data from wildlife can provide new approaches for studying useful correlates of performance that we review. Recent examples include merging traditional resource selection studies with information about resources used at different critical life-history events (e.g. nesting, calving, migration), uncovering habitats that facilitate movement or foraging and, ultimately, comparing resources used through different life-history strategies with those resulting in death. By integrating data from GPS receivers with other animal-borne technologies and combining those data with additional life-history information, we believe understanding the drivers of HPRs will inform animal ecology and improve conservation.
| S-EPMC2894964 | biostudies-other
Project description:The United States National Institutes of Health (NIH) commit significant support to open-source data and software resources in order to foment reproducibility in the biomedical imaging sciences. Here, we report and evaluate a recent product of this commitment: Advanced Neuroimaging Tools (ANTs), which is approaching its 2.0 release. The ANTs open source software library consists of a suite of state-of-the-art image registration, segmentation and template building tools for quantitative morphometric analysis. In this work, we use ANTs to quantify, for the first time, the impact of similarity metrics on the affine and deformable components of a template-based normalization study. We detail the ANTs implementation of three similarity metrics: squared intensity difference, a new and faster cross-correlation, and voxel-wise mutual information. We then use two-fold cross-validation to compare their performance on openly available, manually labeled, T1-weighted MRI brain image data of 40 subjects (UCLA's LPBA40 dataset). We report evaluation results on cortical and whole brain labels for both the affine and deformable components of the registration. Results indicate that the best ANTs methods are competitive with existing brain extraction results (Jaccard=0.958) and cortical labeling approaches. Mutual information affine mapping combined with cross-correlation diffeomorphic mapping gave the best cortical labeling results (Jaccard=0.669±0.022). Furthermore, our two-fold cross-validation allows us to quantify the similarity of templates derived from different subgroups. Our open code, data and evaluation scripts set performance benchmark parameters for this state-of-the-art toolkit. This is the first study to use a consistent transformation framework to provide a reproducible evaluation of the isolated effect of the similarity metric on optimal template construction and brain labeling.
| S-EPMC3065962 | biostudies-literature
Project description:We examined the publication records of a cohort of 168 life scientists in the field of ecology and evolutionary biology to assess gender differences in research performance. Clear discrepancies in publication rate between men and women appear very early in their careers and this has consequences for the subsequent citation of their work. We show that a recently proposed index designed to rank scientists fairly is in fact strongly biased against female researchers, and advocate a modified index to assess men and women on a more equitable basis.
| S-EPMC1762413 | biostudies-literature
Project description:Climate and human management, such as hay harvest, shape grasslands. With both disturbances co-occurring, understanding how these ecosystems respond to these combined drivers may aid in projecting future changes in grasslands. We used an experimental precipitation gradient combined with mimicked acute hay harvest (clipping once a year) to examine (1) whether hay harvest influences precipitation effects on plant performance (cover and height) and (2) the role of inter-specific responses in influencing plant performance. We found that hay harvest reduced the strength of precipitation effects on plant performance through changes in bare-ground soil cover. Species performance were mainly influenced by change in abiotic factors, often responding negatively, as hay harvest increased bare-ground amount. Conversely, altered precipitation without hay harvest promoted plant species performance through abiotic factors change first, followed by biotic. Most species, including the dominant grass Schizachyrium scoparium, increased their performance with greater leaf area index (proxy for canopy structure). Our experiment demonstrates that plant performance responds directly to abiotic factors with hay harvest, but indirectly without hay harvest. Positive effects of increasing precipitation were likely due to microhabitat amelioration and resource acquisition, thus inclusion of hay harvest as a disturbance lessens positive impacts of biotic variables on species performance to climate change.
| S-EPMC8885915 | biostudies-literature
Project description:A bridge role metric model is put forward in this paper. Compared with previous metric models, our solution of a large-scale object-oriented software system as a complex network is inherently more realistic. To acquire nodes and links in an undirected network, a new model that presents the crucial connectivity of a module or the hub instead of only centrality as in previous metric models is presented. Two previous metric models are described for comparison. In addition, it is obvious that the fitting curve between the Bre results and degrees can well be fitted by a power law. The model represents many realistic characteristics of actual software structures, and a hydropower simulation system is taken as an example. This paper makes additional contributions to an accurate understanding of module design of software systems and is expected to be beneficial to software engineering practices.
| S-EPMC4218783 | biostudies-literature