Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Case study in evaluating time series prediction models using the relative mean absolute error.

ABSTRACT: Statistical prediction models inform decision-making processes in many real-world settings. Prior to using predictions in practice, one must rigorously test and validate candidate models to ensure that the proposed predictions have sufficient accuracy to be used in practice. In this paper, we present a framework for evaluating time series predictions that emphasizes computational simplicity and an intuitive interpretation using the relative mean absolute error metric. For a single time series, this metric enables comparisons of candidate model predictions against naïve reference models, a method that can provide useful and standardized performance benchmarks. Additionally, in applications with multiple time series, this framework facilitates comparisons of one or more models' predictive performance across different sets of data. We illustrate the use of this metric with a case study comparing predictions of dengue hemorrhagic fever incidence in two provinces of Thailand. This example demonstrates the utility and interpretability of the relative mean absolute error metric in practice, and underscores the practical advantages of using relative performance metrics when evaluating predictions.

SUBMITTER: Reich NG

PROVIDER: S-EPMC5270768 | biostudies-other | 2016

REPOSITORIES: biostudies-other

ACCESS DATA

Json Xml

Similar Datasets

Design considerations for case series models with exposure onset measurement error.

Project description:The case series model allows for estimation of the relative incidence of events, such as cardiovascular events, within a pre-specified time window after an exposure, such as an infection. The method requires only cases (individuals with events) and controls for all fixed/time-invariant confounders. The measurement error case series model extends the original case series model to handle imperfect data, where the timing of an infection (exposure) is not known precisely. In this work, we propose a method for power/sample size determination for the measurement error case series model. Extensive simulation studies are used to assess the accuracy of the proposed sample size formulas. We also examine the magnitude of the relative loss of power due to exposure onset measurement error, compared with the ideal situation where the time of exposure is measured precisely. To facilitate the design of case series studies, we provide publicly available web-based tools for determining power/sample size for both the measurement error case series model as well as the standard case series model.

| S-EPMC4075338 | biostudies-literature

Decomposition of the mean absolute error (MAE) into systematic and unsystematic components.

Project description:When evaluating the performance of quantitative models, dimensioned errors often are characterized by sums-of-squares measures such as the mean squared error (MSE) or its square root, the root mean squared error (RMSE). In terms of quantifying average error, however, absolute-value-based measures such as the mean absolute error (MAE) are more interpretable than MSE or RMSE. Part of that historical preference for sums-of-squares measures is that they are mathematically amenable to decomposition and one can then form ratios, such as those based on separating MSE into its systematic and unsystematic components. Here, we develop and illustrate a decomposition of MAE into three useful submeasures: (1) bias error, (2) proportionality error, and (3) unsystematic error. This three-part decomposition of MAE is preferable to comparable decompositions of MSE because it provides more straightforward information on the nature of the model-error distribution. We illustrate the properties of our new three-part decomposition using a long-term reconstruction of streamflow for the Upper Colorado River.

| S-EPMC9937461 | biostudies-literature

Glycopeptide Relative Retention Time Prediction

Project description:A re-analysis of published data-dependent acquisition glycoproteomics datasets to demonstrate relative retention time prediction on glycopeptide identification and disambiguation of adduct states.

2020-04-01 | PXD018208 | Pride

Mean Bias in Seasonal Forecast Model and ENSO Prediction Error.

Project description:This study uses retrospective forecasts made using an APEC Climate Center seasonal forecast model to investigate the cause of errors in predicting the amplitude of El Niño Southern Oscillation (ENSO)-driven sea surface temperature variability. When utilizing Bjerknes coupled stability (BJ) index analysis, enhanced errors in ENSO amplitude with forecast lead times are found to be well represented by those in the growth rate estimated by the BJ index. ENSO amplitude forecast errors are most strongly associated with the errors in both the thermocline slope response and surface wind response to forcing over the tropical Pacific, leading to errors in thermocline feedback. This study concludes that upper ocean temperature bias in the equatorial Pacific, which becomes more intense with increasing lead times, is a possible cause of forecast errors in the thermocline feedback and thus in ENSO amplitude.

| S-EPMC5519555 | biostudies-literature

CIS (multiple sclerosis) (case-control) (time-series)

Project description:Clinically isolated syndrome (CIS) refers to the earliest clinical manifestation of multiple sclerosis (MS). Currently there are no prognostic biological markers that accurately predict conversion of CIS to clinically definite MS (CDMS). Furthermore, the earliest molecular events in MS are still unknown. We used microarrays to study gene expression in naïve CD4+ T cells from 37 CIS patients at time of diagnosis and after one year. Supervised machine-learning methods were used to build predictive models of disease conversion. We identified 975 genes whose expression segregated CIS patients into 4 distinct subgroups. A subset of 108 genes further discriminated patients from one of these (group#1) from other CIS patients. Remarkably, 92% of patients from group #1 converted to CDMS within 9 months. Consistent downregulation of TOB1, a critical regulator of cell proliferation, was characteristic of group #1 patients. Decreased TOB1 expression at the RNA and protein levels was also confirmed in experimental autoimmune encephalomyelitis (EAE). Finally, a genetic association was observed between TOB1 variation and MS progression in an independent cohort. These results indicate that CIS patients at high risk of conversion have impaired regulation of T cell quiescence resulting in earlier activation of pathogenic CD4+ cells. Keywords: Disease state: CIS (Clinically Isolated Syndrome - Multiple Sclerosis) versus control. Time series: some patients and controls were resampled approximately one year later

2008-11-26 | GSE13732 | GEO

An alternative trial-level measure for evaluating failure-time surrogate endpoints based on prediction error.

Project description:To validate a failure-time surrogate for an established failure-time clinical endpoint such as overall survival, the meta-analytic approach is commonly used. The standard correlation approach considers two levels: the individual level, with Kendall's τ measuring the rank correlation between the endpoints, and the trial level, with the coefficient of determination R 2 measuring the correlation between the treatment effects on the surrogate and on the final endpoint. However, the estimation of R 2 is not robust with respect to the estimation error of the trial-specific treatment effects. The alternative proposed in this article uses a prediction error based on a measure of the weighted difference between the observed treatment effect on the final endpoint and a model-based predicted effect. The measures can be estimated by cross-validation within the meta-analytic setting or external validation on a set of trials. Several distances are presented, with varying weights, based on the standard error of the observed treatment effect and of its predicted value. A simulation study was conducted under different scenarios, varying the number and the size of the trials, Kendall's τ and R 2 . These measures have been applied to individual patient data from a meta-analysis of trials in advanced/recurrent gastric cancer (20 randomized trials of chemotherapy, 4069 patients). The distance-based measures appeared to be robust with respect to different values of simulation parameters in several scenarios (such as Kendall's τ, size and number of clinical trials). The absolute prediction error can be an alternative to the trial-level R 2 for evaluation of candidate time-to-event surrogates.

| S-EPMC6627581 | biostudies-literature

Measurement Error Case Series Models with Application to Infection-Cardiovascular Risk in OlderPatients on Dialysis.

Project description:Infection and cardiovascular disease are leading causes of hospitalization and death in older patients on dialysis. Our recent work found an increase in the relative incidence of cardiovascular outcomes during the ~ 30 days after infection-related hospitalizations using the case series model, which adjusts for measured and unmeasured baseline confounders. However, a major challenge in modeling/assessing the infection-cardiovascular risk hypothesis is that the exact time of infection, or more generally "exposure," onsets cannot be ascertained based on hospitalization data. Only imprecise markers of the timing of infection onsets are available. Although there is a large literature on measurement error in the predictors in regression modeling, to date there is no work on measurement error on the timing of a time-varying exposure to our knowledge. Thus, we propose a new method, the measurement error case series (MECS) models, to account for measurement error in time-varying exposure onsets. We characterized the general nature of bias resulting from estimation that ignores measurement error and proposed a bias-corrected estimation for the MECS models. We examined in detail the accuracy of the proposed method to estimate the relative incidence. Hospitalization data from United States Renal Data System, which captures nearly all (> 99%) patients with end-stage renal disease in the U.S. over time, is used to illustrate the proposed method. The results suggest that the estimate of the cardiovascular incidence following the 30 days after infections, a period where acute effects of infection on vascular endothelium may be most pronounced, is substantially attenuated in the presence of infection onset measurement error.

| S-EPMC3643015 | biostudies-literature

Nonparametric Prediction in Measurement Error Models.

Project description:Predicting the value of a variable Y corresponding to a future value of an explanatory variable X, based on a sample of previously observed independent data pairs (X(1), Y(1)), …, (X(n), Y(n)) distributed like (X, Y), is very important in statistics. In the error-free case, where X is observed accurately, this problem is strongly related to that of standard regression estimation, since prediction of Y can be achieved via estimation of the regression curve E(Y|X). When the observed X(i)s and the future observation of X are measured with error, prediction is of a quite different nature. Here, if T denotes the future (contaminated) available version of X, prediction of Y can be achieved via estimation of E(Y|T). In practice, estimating E(Y|T) can be quite challenging, as data may be collected under different conditions, making the measurement errors on X(i) and X non-identically distributed. We take up this problem in the nonparametric setting and introduce estimators which allow a highly adaptive approach to smoothing. Reflecting the complexity of the problem, optimal rates of convergence of estimators can vary from the semiparametric n(-1/2) rate to much slower rates that are characteristic of nonparametric problems. Nevertheless, we are able to develop highly adaptive, data-driven methods that achieve very good performance in practice.

| S-EPMC2864536 | biostudies-literature

Robust bootstrap prediction intervals for univariate and multivariate autoregressive time series models.

Project description:The bootstrap procedure has emerged as a general framework to construct prediction intervals for future observations in autoregressive time series models. Such models with outlying data points are standard in real data applications, especially in the field of econometrics. These outlying data points tend to produce high forecast errors, which reduce the forecasting performances of the existing bootstrap prediction intervals calculated based on non-robust estimators. In the univariate and multivariate autoregressive time series, we propose a robust bootstrap algorithm for constructing prediction intervals and forecast regions. The proposed procedure is based on the weighted likelihood estimates and weighted residuals. Its finite sample properties are examined via a series of Monte Carlo studies and two empirical data examples.

| S-EPMC9042141 | biostudies-literature

DaTeR: error-correcting phylogenetic chronograms using relative time constraints.

Project description:MotivationA chronogram is a dated phylogenetic tree whose branch lengths have been scaled to represent time. Such chronograms are computed based on available date estimates (e.g. from dated fossils), which provide absolute time constraints for one or more nodes of an input undated phylogeny, coupled with an appropriate underlying model for evolutionary rates variation along the branches of the phylogeny. However, traditional methods for phylogenetic dating cannot take into account relative time constraints, such as those provided by inferred horizontal transfer events. In many cases, chronograms computed using only absolute time constraints are inconsistent with known relative time constraints.ResultsIn this work, we introduce a new approach, Dating Trees using Relative constraints (DaTeR), for phylogenetic dating that can take into account both absolute and relative time constraints. The key idea is to use existing Bayesian approaches for phylogenetic dating to sample posterior chronograms satisfying desired absolute time constraints, minimally adjust or 'error-correct' these sampled chronograms to satisfy all given relative time constraints, and aggregate across all error-corrected chronograms. DaTeR uses a constrained optimization framework for the error-correction step, finding minimal deviations from previously assigned dates or branch lengths. We applied DaTeR to a biological dataset of 170 Cyanobacterial taxa and a reliable set of 24 transfer-based relative constraints, under six different molecular dating models. Our extensive analysis of this dataset demonstrates that DaTeR is both highly effective and scalable and that its application can significantly improve estimated chronograms.Availability and implementationFreely available from https://compbio.engr.uconn.edu/software/dater/.Supplementary informationSupplementary data are available at Bioinformatics online.

| S-EPMC10075190 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data