Dataset Information

Probabilistic forecasting of replication studies.

ABSTRACT: Throughout the last decade, the so-called replication crisis has stimulated many researchers to conduct large-scale replication projects. With data from four of these projects, we computed probabilistic forecasts of the replication outcomes, which we then evaluated regarding discrimination, calibration and sharpness. A novel model, which can take into account both inflation and heterogeneity of effects, was used and predicted the effect estimate of the replication study with good performance in two of the four data sets. In the other two data sets, predictive performance was still substantially improved compared to the naive model which does not consider inflation and heterogeneity of effects. The results suggest that many of the estimates from the original studies were inflated, possibly caused by publication bias or questionable research practices, and also that some degree of heterogeneity between original and replication effects should be expected. Moreover, the results indicate that the use of statistical significance as the only criterion for replication success may be questionable, since from a predictive viewpoint, non-significant replication results are often compatible with significant results from the original study. The developed statistical methods as well as the data sets are available in the R package ReplicationSuccess.

SUBMITTER: Pawel S

PROVIDER: S-EPMC7176099 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Probabilistic forecasting of replication studies.

Pawel Samuel S Held Leonhard L

PloS one 20200422 4

Throughout the last decade, the so-called replication crisis has stimulated many researchers to conduct large-scale replication projects. With data from four of these projects, we computed probabilistic forecasts of the replication outcomes, which we then evaluated regarding discrimination, calibration and sharpness. A novel model, which can take into account both inflation and heterogeneity of effects, was used and predicted the effect estimate of the replication study with good performance in ...[more]

PMID: 32320420

Similar Datasets

Project description:BackgroundWith enough advanced notice, dengue outbreaks can be mitigated. As a climate-sensitive disease, environmental conditions and past patterns of dengue can be used to make predictions about future outbreak risk. These predictions improve public health planning and decision-making to ultimately reduce the burden of disease. Past approaches to dengue forecasting have used seasonal climate forecasts, but the predictive ability of a system using different lead times in a year-round prediction system has been seldom explored. Moreover, the transition from theoretical to operational systems integrated with disease control activities is rare.Methods and findingsWe introduce an operational seasonal dengue forecasting system for Vietnam where Earth observations, seasonal climate forecasts, and lagged dengue cases are used to drive a superensemble of probabilistic dengue models to predict dengue risk up to 6 months ahead. Bayesian spatiotemporal models were fit to 19 years (2002-2020) of dengue data at the province level across Vietnam. A superensemble of these models then makes probabilistic predictions of dengue incidence at various future time points aligned with key Vietnamese decision and planning deadlines. We demonstrate that the superensemble generates more accurate predictions of dengue incidence than the individual models it incorporates across a suite of time horizons and transmission settings. Using historical data, the superensemble made slightly more accurate predictions (continuous rank probability score [CRPS] = 66.8, 95% CI 60.6-148.0) than a baseline model which forecasts the same incidence rate every month (CRPS = 79.4, 95% CI 78.5-80.5) at lead times of 1 to 3 months, albeit with larger uncertainty. The outbreak detection capability of the superensemble was considerably larger (69%) than that of the baseline model (54.5%). Predictions were most accurate in southern Vietnam, an area that experiences semi-regular seasonal dengue transmission. The system also demonstrated added value across multiple areas compared to previous practice of not using a forecast. We use the system to make a prospective prediction for dengue incidence in Vietnam for the period May to October 2020. Prospective predictions made with the superensemble were slightly more accurate (CRPS = 110, 95% CI 102-575) than those made with the baseline model (CRPS = 125, 95% CI 120-168) but had larger uncertainty. Finally, we propose a framework for the evaluation of probabilistic predictions. Despite the demonstrated value of our forecasting system, the approach is limited by the consistency of the dengue case data, as well as the lack of publicly available, continuous, and long-term data sets on mosquito control efforts and serotype-specific case data.ConclusionsThis study shows that by combining detailed Earth observation data, seasonal climate forecasts, and state-of-the-art models, dengue outbreaks can be predicted across a broad range of settings, with enough lead time to meaningfully inform dengue control. While our system omits some important variables not currently available at a subnational scale, the majority of past outbreaks could be predicted up to 3 months ahead. Over the next 2 years, the system will be prospectively evaluated and, if successful, potentially extended to other areas and other climate-sensitive disease systems.

Project description:PurposeTo probabilistically forecast needed anti-vascular endothelial growth factor (anti-VEGF) treatment frequency using volumetric spectral domain-optical coherence tomography (SD-OCT) biomarkers in neovascular age-related macular degeneration from real-world settings.MethodsSD-OCT volume scans were segmented with a custom deep-learning-based analysis pipeline. Retinal thickness and reflectivity values were extracted for the central and the four inner Early Treatment Diabetic Retinopathy Study (ETDRS) subfields for six retinal layers (inner retina, outer nuclear layer, inner segments [IS], outer segments [OS], retinal pigment epithelium-drusen complex [RPEDC] and the choroid). Machine-learning models were probed to predict the anti-VEGF treatment frequency within the next 12 months. Probabilistic forecasting was performed using natural gradient boosting (NGBoost), which outputs a full probability distribution. The mean absolute error (MAE) between the predicted versus actual anti-VEGF treatment frequency was the primary outcome measure.ResultsIn a total of 138 visits of 99 eyes with neovascular AMD (96 patients) from two clinical centers, the prediction of future anti-VEGF treatment frequency was observed with an accuracy (MAE [95% confidence interval]) of 2.60 injections/year [2.25-2.96] (R2 = 0.390) using random forest regression and 2.66 injections/year [2.31-3.01] (R2 = 0.094) using NGBoost, respectively. Prediction intervals were well calibrated and reflected the true uncertainty of NGBoost-based predictions. Standard deviation of RPEDC-thickness in the central ETDRS-subfield constituted an important predictor across models.ConclusionsThe proposed, fully automated pipeline enables probabilistic forecasting of future anti-VEGF treatment frequency in real-world settings.Translational relevancePrediction of a probability distribution allows the physician to inspect the underlying uncertainty. Predictive uncertainty estimates are essential to highlight cases where human-inspection and/or reversion to a fallback alternative is warranted.

Dataset Information

Probabilistic forecasting of replication studies.

Publications

Probabilistic forecasting of replication studies.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets