Dataset Information

Testing for ontological errors in probabilistic forecasting models of natural systems.

ABSTRACT: Probabilistic forecasting models describe the aleatory variability of natural systems as well as our epistemic uncertainty about how the systems work. Testing a model against observations exposes ontological errors in the representation of a system and its uncertainties. We clarify several conceptual issues regarding the testing of probabilistic forecasting models for ontological errors: the ambiguity of the aleatory/epistemic dichotomy, the quantification of uncertainties as degrees of belief, the interplay between Bayesian and frequentist methods, and the scientific pathway for capturing predictability. We show that testability of the ontological null hypothesis derives from an experimental concept, external to the model, that identifies collections of data, observed and not yet observed, that are judged to be exchangeable when conditioned on a set of explanatory variables. These conditional exchangeability judgments specify observations with well-defined frequencies. Any model predicting these behaviors can thus be tested for ontological error by frequentist methods; e.g., using P values. In the forecasting problem, prior predictive model checking, rather than posterior predictive checking, is desirable because it provides more severe tests. We illustrate experimental concepts using examples from probabilistic seismic hazard analysis. Severe testing of a model under an appropriate set of experimental concepts is the key to model validation, in which we seek to know whether a model replicates the data-generating process well enough to be sufficiently reliable for some useful purpose, such as long-term seismic forecasting. Pessimistic views of system predictability fail to recognize the power of this methodology in separating predictable behaviors from those that are not.

SUBMITTER: Marzocchi W

PROVIDER: S-EPMC4143071 | biostudies-literature | 2014 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Testing for ontological errors in probabilistic forecasting models of natural systems.

Marzocchi Warner W Jordan Thomas H TH

Proceedings of the National Academy of Sciences of the United States of America 20140805 33

Probabilistic forecasting models describe the aleatory variability of natural systems as well as our epistemic uncertainty about how the systems work. Testing a model against observations exposes ontological errors in the representation of a system and its uncertainties. We clarify several conceptual issues regarding the testing of probabilistic forecasting models for ontological errors: the ambiguity of the aleatory/epistemic dichotomy, the quantification of uncertainties as degrees of belief, ...[more]

PMID: 25097265

Similar Datasets

Project description:BackgroundWith enough advanced notice, dengue outbreaks can be mitigated. As a climate-sensitive disease, environmental conditions and past patterns of dengue can be used to make predictions about future outbreak risk. These predictions improve public health planning and decision-making to ultimately reduce the burden of disease. Past approaches to dengue forecasting have used seasonal climate forecasts, but the predictive ability of a system using different lead times in a year-round prediction system has been seldom explored. Moreover, the transition from theoretical to operational systems integrated with disease control activities is rare.Methods and findingsWe introduce an operational seasonal dengue forecasting system for Vietnam where Earth observations, seasonal climate forecasts, and lagged dengue cases are used to drive a superensemble of probabilistic dengue models to predict dengue risk up to 6 months ahead. Bayesian spatiotemporal models were fit to 19 years (2002-2020) of dengue data at the province level across Vietnam. A superensemble of these models then makes probabilistic predictions of dengue incidence at various future time points aligned with key Vietnamese decision and planning deadlines. We demonstrate that the superensemble generates more accurate predictions of dengue incidence than the individual models it incorporates across a suite of time horizons and transmission settings. Using historical data, the superensemble made slightly more accurate predictions (continuous rank probability score [CRPS] = 66.8, 95% CI 60.6-148.0) than a baseline model which forecasts the same incidence rate every month (CRPS = 79.4, 95% CI 78.5-80.5) at lead times of 1 to 3 months, albeit with larger uncertainty. The outbreak detection capability of the superensemble was considerably larger (69%) than that of the baseline model (54.5%). Predictions were most accurate in southern Vietnam, an area that experiences semi-regular seasonal dengue transmission. The system also demonstrated added value across multiple areas compared to previous practice of not using a forecast. We use the system to make a prospective prediction for dengue incidence in Vietnam for the period May to October 2020. Prospective predictions made with the superensemble were slightly more accurate (CRPS = 110, 95% CI 102-575) than those made with the baseline model (CRPS = 125, 95% CI 120-168) but had larger uncertainty. Finally, we propose a framework for the evaluation of probabilistic predictions. Despite the demonstrated value of our forecasting system, the approach is limited by the consistency of the dengue case data, as well as the lack of publicly available, continuous, and long-term data sets on mosquito control efforts and serotype-specific case data.ConclusionsThis study shows that by combining detailed Earth observation data, seasonal climate forecasts, and state-of-the-art models, dengue outbreaks can be predicted across a broad range of settings, with enough lead time to meaningfully inform dengue control. While our system omits some important variables not currently available at a subnational scale, the majority of past outbreaks could be predicted up to 3 months ahead. Over the next 2 years, the system will be prospectively evaluated and, if successful, potentially extended to other areas and other climate-sensitive disease systems.

Project description:Social scientists have postulated that the discrepancy between achievements and expectations affects individuals' subjective well-being. Still, little has been done to qualify and quantify such a psychological effect. Our empirical analysis assesses the consequences of positive and negative affective forecasting errors-the difference between realized and expected subjective well-being-on the subsequent level of subjective well-being.We use longitudinal data on a representative sample of 13,431 individuals from the German Socio-Economic Panel. In our sample, 52% of individuals are females, average age is 43 years, average years of education is 11.4 and 27% of our sample lives in East Germany. Subjective well-being (measured by self-reported life satisfaction) is assessed on a 0-10 discrete scale and its sample average is equal to 6.75 points.We develop a simple theoretical framework to assess the consequences of positive and negative affective forecasting errors-the difference between realized and expected subjective well-being-on the subsequent level of subjective well-being, properly accounting for the endogenous adjustment of expectations to positive and negative affective forecasting errors, and use it to derive testable predictions. Given the theoretical framework, we estimate two panel-data equations, the first depicting the association between positive and negative affective forecasting errors and the successive level of subjective well-being and the second describing the correlation between subjective well-being expectations for the future and hedonic failures and successes. Our models control for individual fixed effects and a large battery of time-varying demographic characteristics, health and socio-economic status.While surpassing expectations is uncorrelated with subjective well-being, failing to match expectations is negatively associated with subsequent realizations of subjective well-being. Expectations are positively (negatively) correlated to positive (negative) forecasting errors. We speculate that in the first case the positive adjustment in expectations is strong enough to cancel out the potential positive effects on subjective well-being of beaten expectations, while in the second case it is not, and individuals persistently bear the negative emotional consequences of not achieving expectations.

Dataset Information

Testing for ontological errors in probabilistic forecasting models of natural systems.

Publications

Testing for ontological errors in probabilistic forecasting models of natural systems.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure