Improving probabilistic infectious disease forecasting through coherence.
Ontology highlight
ABSTRACT:
SUBMITTER: Gibson GC
PROVIDER: S-EPMC7837472 | biostudies-literature | 2021 Jan
REPOSITORIES: biostudies-literature
Publications
Improving probabilistic infectious disease forecasting through coherence.
Gibson Graham Casey GC Moran Kelly R KR Reich Nicholas G NG Osthus Dave D
PLoS computational biology 20210106 1
With an estimated $10.4 billion in medical costs and 31.4 million outpatient visits each year, influenza poses a serious burden of disease in the United States. To provide insights and advance warning into the spread of influenza, the U.S. Centers for Disease Control and Prevention (CDC) runs a challenge for forecasting weighted influenza-like illness (wILI) at the national and regional level. Many models produce independent forecasts for each geographical unit, ignoring the constraint that the ...[more]
Similar Datasets
Project description:Epidemiological models are powerful tools for evaluating scenarios and visualizing patterns of disease spread, especially when comparing intervention strategies. However, the technical skill required to synthesize and operate computational models frequently renders them beyond the command of the stakeholders who are most impacted by the results. Participatory modelling (PM) strives to restructure the power relationship between modellers and the stakeholders who rely on model insights by involving these stakeholders directly in model development and application; yet, a systematic literature review indicates little adoption of these techniques in epidemiology, especially plant epidemiology. We investigate the potential for PM to integrate stakeholder and researcher knowledge, using Phytophthora ramorum and the resulting sudden oak death disease as a case study. Recent introduction of a novel strain (European 1 or EU1) in southwestern Oregon has prompted significant concern and presents an opportunity for coordinated management to minimize regional pathogen impacts. Using a PM framework, we worked with local stakeholders to develop an interactive forecasting tool for evaluating landscape-scale control strategies. We find that model co-development has great potential to empower stakeholders in the design, development and application of epidemiological models for disease control. This article is part of the theme issue 'Modelling infectious disease outbreaks in humans, animals and plants: epidemic forecasting and control'. This theme issue is linked with the earlier issue 'Modelling infectious disease outbreaks in humans, animals and plants: approaches and important themes'.
| S-EPMC6558554 | biostudies-literature
Project description:Throughout the last decade, the so-called replication crisis has stimulated many researchers to conduct large-scale replication projects. With data from four of these projects, we computed probabilistic forecasts of the replication outcomes, which we then evaluated regarding discrimination, calibration and sharpness. A novel model, which can take into account both inflation and heterogeneity of effects, was used and predicted the effect estimate of the replication study with good performance in two of the four data sets. In the other two data sets, predictive performance was still substantially improved compared to the naive model which does not consider inflation and heterogeneity of effects. The results suggest that many of the estimates from the original studies were inflated, possibly caused by publication bias or questionable research practices, and also that some degree of heterogeneity between original and replication effects should be expected. Moreover, the results indicate that the use of statistical significance as the only criterion for replication success may be questionable, since from a predictive viewpoint, non-significant replication results are often compatible with significant results from the original study. The developed statistical methods as well as the data sets are available in the R package ReplicationSuccess.
| S-EPMC7176099 | biostudies-literature
Project description:BackgroundFew researchers have assessed the relationships between socioeconomic inequality and infectious disease outbreaks at the population level globally. We use a socioeconomic model to forecast national annual rates of infectious disease outbreaks.MethodsWe constructed a multivariate mixed-effects Poisson model of the number of times a given country was the origin of an outbreak in a given year. The dataset included 389 outbreaks of international concern reported in the World Health Organization's Disease Outbreak News from 1996 to 2008. The initial full model included 9 socioeconomic variables related to education, poverty, population health, urbanization, health infrastructure, gender equality, communication, transportation, and democracy, and 1 composite index. Population, latitude, and elevation were included as potential confounders. The initial model was pared down to a final model by a backwards elimination procedure. The dependent and independent variables were lagged by 2 years to allow for forecasting future rates.ResultsAmong the socioeconomic variables tested, the final model included child measles immunization rate and telephone line density. The Democratic Republic of Congo, China, and Brazil were predicted to be at the highest risk for outbreaks in 2010, and Colombia and Indonesia were predicted to have the highest percentage of increase in their risk compared to their average over 1996-2008.ConclusionsUnderstanding socioeconomic factors could help improve the understanding of outbreak risk. The inclusion of the measles immunization variable suggests that there is a fundamental basis in ensuring adequate public health capacity. Increased vigilance and expanding public health capacity should be prioritized in the projected high-risk regions.
| S-EPMC3552528 | biostudies-literature
Project description:Anthropogenic warming has led to an unprecedented year-round reduction in Arctic sea ice extent. This has far-reaching consequences for indigenous and local communities, polar ecosystems, and global climate, motivating the need for accurate seasonal sea ice forecasts. While physics-based dynamical models can successfully forecast sea ice concentration several weeks ahead, they struggle to outperform simple statistical benchmarks at longer lead times. We present a probabilistic, deep learning sea ice forecasting system, IceNet. The system has been trained on climate simulations and observational data to forecast the next 6 months of monthly-averaged sea ice concentration maps. We show that IceNet advances the range of accurate sea ice forecasts, outperforming a state-of-the-art dynamical model in seasonal forecasts of summer sea ice, particularly for extreme sea ice events. This step-change in sea ice forecasting ability brings us closer to conservation tools that mitigate risks associated with rapid sea ice loss.
| S-EPMC8390499 | biostudies-literature
Project description:Real-time forecasts of infectious diseases can help public health planning, especially during outbreaks. If forecasts are generated from mechanistic models, they can be further used to target resources or to compare the impact of possible interventions. However, paremeterising such models is often difficult in real time, when information on behavioural changes, interventions and routes of transmission are not readily available. Here, we present a semi-mechanistic model of infectious disease dynamics that was used in real time during the 2013-2016 West African Ebola epidemic, and show fits to a Ebola Forecasting Challenge conducted in late 2015 with simulated data mimicking the true epidemic. We assess the performance of the model in different situations and identify strengths and shortcomings of our approach. Models such as the one presented here which combine the power of mechanistic models with the flexibility to include uncertainty about the precise outbreak dynamics may be an important tool in combating future outbreaks.
| S-EPMC5871642 | biostudies-literature
Project description:A wide range of research has promised new tools for forecasting infectious disease dynamics, but little of that research is currently being applied in practice, because tools do not address key public health needs, do not produce probabilistic forecasts, have not been evaluated on external data, or do not provide sufficient forecast skill to be useful. We developed an open collaborative forecasting challenge to assess probabilistic forecasts for seasonal epidemics of dengue, a major global public health problem. Sixteen teams used a variety of methods and data to generate forecasts for 3 epidemiological targets (peak incidence, the week of the peak, and total incidence) over 8 dengue seasons in Iquitos, Peru and San Juan, Puerto Rico. Forecast skill was highly variable across teams and targets. While numerous forecasts showed high skill for midseason situational awareness, early season skill was low, and skill was generally lowest for high incidence seasons, those for which forecasts would be most valuable. A comparison of modeling approaches revealed that average forecast skill was lower for models including biologically meaningful data and mechanisms and that both multimodel and multiteam ensemble forecasts consistently outperformed individual model forecasts. Leveraging these insights, data, and the forecasting framework will be critical to improve forecast skill and the application of forecasts in real time for epidemic preparedness and response. Moreover, key components of this project-integration with public health needs, a common forecasting framework, shared and standardized data, and open participation-can help advance infectious disease forecasting beyond dengue.
| S-EPMC6883829 | biostudies-literature
Project description:Temporal hierarchies have been widely used during the past few years as they are capable to provide more accurate coherent forecasts at different planning horizons. However, they still display some limitations, being mainly subject to the forecasting methods used for generating the base forecasts and the particularities of the examined series. This paper deals with such limitations by considering three different strategies: (i) combining forecasts of multiple methods, (ii) applying bias adjustments and (iii) selectively implementing temporal hierarchies to avoid seasonal shrinkage. The proposed strategies can be applied either separately or simultaneously, being complements to the method considered for reconciling the base forecasts and completely independent from each other. Their effect is evaluated using the monthly series of the M and M3 competitions. The results are very promising, displaying lots of potential for improving the performance of temporal hierarchies, both in terms of accuracy and bias.
| S-EPMC6776316 | biostudies-literature
Project description:Probabilistic forecasting models describe the aleatory variability of natural systems as well as our epistemic uncertainty about how the systems work. Testing a model against observations exposes ontological errors in the representation of a system and its uncertainties. We clarify several conceptual issues regarding the testing of probabilistic forecasting models for ontological errors: the ambiguity of the aleatory/epistemic dichotomy, the quantification of uncertainties as degrees of belief, the interplay between Bayesian and frequentist methods, and the scientific pathway for capturing predictability. We show that testability of the ontological null hypothesis derives from an experimental concept, external to the model, that identifies collections of data, observed and not yet observed, that are judged to be exchangeable when conditioned on a set of explanatory variables. These conditional exchangeability judgments specify observations with well-defined frequencies. Any model predicting these behaviors can thus be tested for ontological error by frequentist methods; e.g., using P values. In the forecasting problem, prior predictive model checking, rather than posterior predictive checking, is desirable because it provides more severe tests. We illustrate experimental concepts using examples from probabilistic seismic hazard analysis. Severe testing of a model under an appropriate set of experimental concepts is the key to model validation, in which we seek to know whether a model replicates the data-generating process well enough to be sufficiently reliable for some useful purpose, such as long-term seismic forecasting. Pessimistic views of system predictability fail to recognize the power of this methodology in separating predictable behaviors from those that are not.
| S-EPMC4143071 | biostudies-literature
Project description:BackgroundWith enough advanced notice, dengue outbreaks can be mitigated. As a climate-sensitive disease, environmental conditions and past patterns of dengue can be used to make predictions about future outbreak risk. These predictions improve public health planning and decision-making to ultimately reduce the burden of disease. Past approaches to dengue forecasting have used seasonal climate forecasts, but the predictive ability of a system using different lead times in a year-round prediction system has been seldom explored. Moreover, the transition from theoretical to operational systems integrated with disease control activities is rare.Methods and findingsWe introduce an operational seasonal dengue forecasting system for Vietnam where Earth observations, seasonal climate forecasts, and lagged dengue cases are used to drive a superensemble of probabilistic dengue models to predict dengue risk up to 6 months ahead. Bayesian spatiotemporal models were fit to 19 years (2002-2020) of dengue data at the province level across Vietnam. A superensemble of these models then makes probabilistic predictions of dengue incidence at various future time points aligned with key Vietnamese decision and planning deadlines. We demonstrate that the superensemble generates more accurate predictions of dengue incidence than the individual models it incorporates across a suite of time horizons and transmission settings. Using historical data, the superensemble made slightly more accurate predictions (continuous rank probability score [CRPS] = 66.8, 95% CI 60.6-148.0) than a baseline model which forecasts the same incidence rate every month (CRPS = 79.4, 95% CI 78.5-80.5) at lead times of 1 to 3 months, albeit with larger uncertainty. The outbreak detection capability of the superensemble was considerably larger (69%) than that of the baseline model (54.5%). Predictions were most accurate in southern Vietnam, an area that experiences semi-regular seasonal dengue transmission. The system also demonstrated added value across multiple areas compared to previous practice of not using a forecast. We use the system to make a prospective prediction for dengue incidence in Vietnam for the period May to October 2020. Prospective predictions made with the superensemble were slightly more accurate (CRPS = 110, 95% CI 102-575) than those made with the baseline model (CRPS = 125, 95% CI 120-168) but had larger uncertainty. Finally, we propose a framework for the evaluation of probabilistic predictions. Despite the demonstrated value of our forecasting system, the approach is limited by the consistency of the dengue case data, as well as the lack of publicly available, continuous, and long-term data sets on mosquito control efforts and serotype-specific case data.ConclusionsThis study shows that by combining detailed Earth observation data, seasonal climate forecasts, and state-of-the-art models, dengue outbreaks can be predicted across a broad range of settings, with enough lead time to meaningfully inform dengue control. While our system omits some important variables not currently available at a subnational scale, the majority of past outbreaks could be predicted up to 3 months ahead. Over the next 2 years, the system will be prospectively evaluated and, if successful, potentially extended to other areas and other climate-sensitive disease systems.
| S-EPMC7971894 | biostudies-literature
Project description:PurposeTo probabilistically forecast needed anti-vascular endothelial growth factor (anti-VEGF) treatment frequency using volumetric spectral domain-optical coherence tomography (SD-OCT) biomarkers in neovascular age-related macular degeneration from real-world settings.MethodsSD-OCT volume scans were segmented with a custom deep-learning-based analysis pipeline. Retinal thickness and reflectivity values were extracted for the central and the four inner Early Treatment Diabetic Retinopathy Study (ETDRS) subfields for six retinal layers (inner retina, outer nuclear layer, inner segments [IS], outer segments [OS], retinal pigment epithelium-drusen complex [RPEDC] and the choroid). Machine-learning models were probed to predict the anti-VEGF treatment frequency within the next 12 months. Probabilistic forecasting was performed using natural gradient boosting (NGBoost), which outputs a full probability distribution. The mean absolute error (MAE) between the predicted versus actual anti-VEGF treatment frequency was the primary outcome measure.ResultsIn a total of 138 visits of 99 eyes with neovascular AMD (96 patients) from two clinical centers, the prediction of future anti-VEGF treatment frequency was observed with an accuracy (MAE [95% confidence interval]) of 2.60 injections/year [2.25-2.96] (R2 = 0.390) using random forest regression and 2.66 injections/year [2.31-3.01] (R2 = 0.094) using NGBoost, respectively. Prediction intervals were well calibrated and reflected the true uncertainty of NGBoost-based predictions. Standard deviation of RPEDC-thickness in the central ETDRS-subfield constituted an important predictor across models.ConclusionsThe proposed, fully automated pipeline enables probabilistic forecasting of future anti-VEGF treatment frequency in real-world settings.Translational relevancePrediction of a probability distribution allows the physician to inspect the underlying uncertainty. Predictive uncertainty estimates are essential to highlight cases where human-inspection and/or reversion to a fallback alternative is warranted.
| S-EPMC8254013 | biostudies-literature