Fitting dynamic models to epidemic outbreaks with quantified uncertainty: A Primer for parameter uncertainty, identifiability, and forecasts.
Ontology highlight
ABSTRACT:
SUBMITTER: Chowell G
PROVIDER: S-EPMC5726591 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Deterministic dynamic causal modeling (DCM) for fMRI data is a sophisticated approach to analyse effective connectivity in terms of directed interactions between brain regions of interest. To date it is difficult to know if acquired fMRI data will yield precise estimation of DCM parameters. Focusing on parameter identifiability, an important prerequisite for research questions on directed connectivity, we present an approach inferring if parameters of an envisaged DCM are identifiable based on information from fMRI data. With the freely available "attention to motion" dataset, we investigate identifiability of two DCMs and show how different imaging specifications impact on identifiability. We used the profile likelihood, which has successfully been applied in systems biology, to assess the identifiability of parameters in a DCM with specified scanning parameters. Parameters are identifiable when minima of the profile likelihood as well as finite confidence intervals for the parameters exist. Intermediate epoch duration, shorter TR and longer session duration generally increased the information content in the data and thus improved identifiability. Irrespective of biological factors such as size and location of a region, attention should be paid to densely interconnected regions in a DCM, as those seem to be prone to non-identifiability. Our approach, available in the DCMident toolbox, enables to judge if the parameters of an envisaged DCM are sufficiently determined by underlying data without priors as opposed to primarily reflecting the Bayesian priors in a SPM-DCM. Assessments with the DCMident toolbox prior to a study will lead to improved identifiability of the parameters and thus might prevent suboptimal data acquisition. Thus, the toolbox can be used as a preprocessing step to provide immediate statements on parameter identifiability.
| S-EPMC4335185 | biostudies-literature
Project description:BackgroundModels for complex biological systems may involve a large number of parameters. It may well be that some of these parameters cannot be derived from observed data via regression techniques. Such parameters are said to be unidentifiable, the remaining parameters being identifiable. Closely related to this idea is that of redundancy, that a set of parameters can be expressed in terms of some smaller set. Before data is analysed it is critical to determine which model parameters are identifiable or redundant to avoid ill-defined and poorly convergent regression.Methodology/principal findingsIn this paper we outline general considerations on parameter identifiability, and introduce the notion of weak local identifiability and gradient weak local identifiability. These are based on local properties of the likelihood, in particular the rank of the Hessian matrix. We relate these to the notions of parameter identifiability and redundancy previously introduced by Rothenberg (Econometrica 39 (1971) 577-591) and Catchpole and Morgan (Biometrika 84 (1997) 187-196). Within the widely used exponential family, parameter irredundancy, local identifiability, gradient weak local identifiability and weak local identifiability are shown to be largely equivalent. We consider applications to a recently developed class of cancer models of Little and Wright (Math Biosciences 183 (2003) 111-134) and Little et al. (J Theoret Biol 254 (2008) 229-238) that generalize a large number of other recently used quasi-biological cancer models.Conclusions/significanceWe have shown that the previously developed concepts of parameter local identifiability and redundancy are closely related to the apparently weaker properties of weak local identifiability and gradient weak local identifiability--within the widely used exponential family these concepts largely coincide.
| S-EPMC2811744 | biostudies-literature
Project description:Issues of parameter identifiability of routinely used pharmacodynamics models are considered in this paper. The structural identifiability of 16 commonly applied pharmacodynamic model structures was analyzed analytically, using the input-output approach. Both fixed-effects versions (non-population, no between-subject variability) and mixed-effects versions (population, including between-subject variability) of each model structure were analyzed. All models were found to be structurally globally identifiable under conditions of fixing either one of two particular parameters. Furthermore, an example was constructed to illustrate the importance of sufficient data quality and show that structural identifiability is a prerequisite, but not a guarantee, for successful parameter estimation and practical parameter identifiability. This analysis was performed by generating artificial data of varying quality to a structurally identifiable model with known true parameter values, followed by re-estimation of the parameter values. In addition, to show the benefit of including structural identifiability as part of model development, a case study was performed applying an unidentifiable model to real experimental data. This case study shows how performing such an analysis prior to parameter estimation can improve the parameter estimation process and model performance. Finally, an unidentifiable model was fitted to simulated data using multiple initial parameter values, resulting in highly different estimated uncertainties. This example shows that although the standard errors of the parameter estimates often indicate a structural identifiability issue, reasonably "good" standard errors may sometimes mask unidentifiability issues.
| S-EPMC5136565 | biostudies-literature
Project description:In recent years, a number of systems capable of predicting future infectious disease incidence have been developed. As more of these systems are operationalized, it is important that the forecasts generated by these different approaches be formally reconciled so that individual forecast error and bias are reduced. Here we present a first example of such multi-system, or superensemble, forecast. We develop three distinct systems for predicting dengue, which are applied retrospectively to forecast outbreak characteristics in San Juan, Puerto Rico. We then use Bayesian averaging methods to combine the predictions from these systems and create superensemble forecasts. We demonstrate that on average, the superensemble approach produces more accurate forecasts than those made from any of the individual forecasting systems.
| S-EPMC5095208 | biostudies-literature
Project description:Stochastic transmission dynamic models are needed to quantify the uncertainty in estimates and predictions during outbreaks of infectious diseases. We previously developed a calibration method for stochastic epidemic compartmental models, called Multiple Shooting for Stochastic Systems (MSS), and demonstrated its competitive performance against a number of existing state-of-the-art calibration methods. The existing MSS method, however, lacks a mechanism against filter degeneracy, a phenomenon that results in parameter posterior distributions that are weighted heavily around a single value. As such, when filter degeneracy occurs, the posterior distributions of parameter estimates will not yield reliable credible or prediction intervals for parameter estimates and predictions. In this work, we extend the MSS method by evaluating and incorporating two resampling techniques to detect and resolve filter degeneracy. Using simulation experiments, we demonstrate that an extended MSS method produces credible and prediction intervals with desired coverage in estimating key epidemic parameters (e.g. mean duration of infectiousness and R0) and short- and long-term predictions (e.g. one and three-week forecasts, timing and number of cases at the epidemic peak, and final epidemic size). Applying the extended MSS approach to a humidity-based stochastic compartmental influenza model, we were able to accurately predict influenza-like illness activity reported by U.S. Centers for Disease Control and Prevention from 10 regions as well as city-level influenza activity using real-time, city-specific Google search query data from 119 U.S. cities between 2003 and 2014.
| S-EPMC6517086 | biostudies-literature
Project description:As the Arctic region moves into uncharted territory under a warming climate, it is important to refine the terrestrial biosphere models (TBMs) that help us understand and predict change. One fundamental uncertainty in TBMs relates to model parameters, configuration variables internal to the model whose value can be estimated from data. We incorporate a version of the Terrestrial Ecosystem Model (TEM) developed for arctic ecosystems into the Predictive Ecosystem Analyzer (PEcAn) framework. PEcAn treats model parameters as probability distributions, estimates parameters based on a synthesis of available field data, and then quantifies both model sensitivity and uncertainty to a given parameter or suite of parameters. We examined how variation in 21 parameters in the equation for gross primary production influenced model sensitivity and uncertainty in terms of two carbon fluxes (net primary productivity and heterotrophic respiration) and two carbon (C) pools (vegetation C and soil C). We set up different parameterizations of TEM across a range of tundra types (tussock tundra, heath tundra, wet sedge tundra, and shrub tundra) in northern Alaska, along a latitudinal transect extending from the coastal plain near Utqiaġvik to the southern foothills of the Brooks Range, to the Seward Peninsula. TEM was most sensitive to parameters related to the temperature regulation of photosynthesis. Model uncertainty was mostly due to parameters related to leaf area, temperature regulation of photosynthesis, and the stomatal responses to ambient light conditions. Our analysis also showed that sensitivity and uncertainty to a given parameter varied spatially. At some sites, model sensitivity and uncertainty tended to be connected to a wider range of parameters, underlining the importance of assessing tundra community processes across environmental gradients or geographic locations. Generally, across sites, the flux of net primary productivity (NPP) and pool of vegetation C had about equal uncertainty, while heterotrophic respiration had higher uncertainty than the pool of soil C. Our study illustrates the complexity inherent in evaluating parameter uncertainty across highly heterogeneous arctic tundra plant communities. It also provides a framework for iteratively testing how newly collected field data related to key parameters may result in more effective forecasting of Arctic change.
| S-EPMC9285828 | biostudies-literature
Project description:BackgroundMathematical modeling is now frequently used in outbreak investigations to understand underlying mechanisms of infectious disease dynamics, assess patterns in epidemiological data, and forecast the trajectory of epidemics. However, the successful application of mathematical models to guide public health interventions lies in the ability to reliably estimate model parameters and their corresponding uncertainty. Here, we present and illustrate a simple computational method for assessing parameter identifiability in compartmental epidemic models.MethodsWe describe a parametric bootstrap approach to generate simulated data from dynamical systems to quantify parameter uncertainty and identifiability. We calculate confidence intervals and mean squared error of estimated parameter distributions to assess parameter identifiability. To demonstrate this approach, we begin with a low-complexity SEIR model and work through examples of increasingly more complex compartmental models that correspond with applications to pandemic influenza, Ebola, and Zika.ResultsOverall, parameter identifiability issues are more likely to arise with more complex models (based on number of equations/states and parameters). As the number of parameters being jointly estimated increases, the uncertainty surrounding estimated parameters tends to increase, on average, as well. We found that, in most cases, R0 is often robust to parameter identifiability issues affecting individual parameters in the model. Despite large confidence intervals and higher mean squared error of other individual model parameters, R0 can still be estimated with precision and accuracy.ConclusionsBecause public health policies can be influenced by results of mathematical modeling studies, it is important to conduct parameter identifiability analyses prior to fitting the models to available data and to report parameter estimates with quantified uncertainty. The method described is helpful in these regards and enhances the essential toolkit for conducting model-based inferences using compartmental dynamic models.
| S-EPMC6332839 | biostudies-literature
Project description:The success of model-based methods in phylogenetics has motivated much research aimed at generating new, biologically informative models. This new computer-intensive approach to phylogenetics demands validation studies and sound measures of performance. To date there has been little practical guidance available as to when and why the parameters in a particular model can be identified reliably. Here, we illustrate how Data Cloning (DC), a recently developed methodology to compute the maximum likelihood estimates along with their asymptotic variance, can be used to diagnose structural parameter nonidentifiability (NI) and distinguish it from other parameter estimability problems, including when parameters are structurally identifiable, but are not estimable in a given data set (INE), and when parameters are identifiable, and estimable, but only weakly so (WE). The application of the DC theorem uses well-known and widely used Bayesian computational techniques. With the DC approach, practitioners can use Bayesian phylogenetics software to diagnose nonidentifiability. Theoreticians and practitioners alike now have a powerful, yet simple tool to detect nonidentifiability while investigating complex modeling scenarios, where getting closed-form expressions in a probabilistic study is complicated. Furthermore, here we also show how DC can be used as a tool to examine and eliminate the influence of the priors, in particular if the process of prior elicitation is not straightforward. Finally, when applied to phylogenetic inference, DC can be used to study at least two important statistical questions: assessing identifiability of discrete parameters, like the tree topology, and developing efficient sampling methods for computationally expensive posterior densities.
| S-EPMC3478565 | biostudies-literature
Project description:BackgroundOver the past few decades, numerous forecasting methods have been proposed in the field of epidemic forecasting. Such methods can be classified into different categories such as deterministic vs. probabilistic, comparative methods vs. generative methods, and so on. In some of the more popular comparative methods, researchers compare observed epidemiological data from the early stages of an outbreak with the output of proposed models to forecast the future trend and prevalence of the pandemic. A significant problem in this area is the lack of standard well-defined evaluation measures to select the best algorithm among different ones, as well as for selecting the best possible configuration for a particular algorithm.ResultsIn this paper we present an evaluation framework which allows for combining different features, error measures, and ranking schema to evaluate forecasts. We describe the various epidemic features (Epi-features) included to characterize the output of forecasting methods and provide suitable error measures that could be used to evaluate the accuracy of the methods with respect to these Epi-features. We focus on long-term predictions rather than short-term forecasting and demonstrate the utility of the framework by evaluating six forecasting methods for predicting influenza in the United States. Our results demonstrate that different error measures lead to different rankings even for a single Epi-feature. Further, our experimental analyses show that no single method dominates the rest in predicting all Epi-features when evaluated across error measures. As an alternative, we provide various Consensus Ranking schema that summarize individual rankings, thus accounting for different error measures. Since each Epi-feature presents a different aspect of the epidemic, multiple methods need to be combined to provide a comprehensive forecast. Thus we call for a more nuanced approach while evaluating epidemic forecasts and we believe that a comprehensive evaluation framework, as presented in this paper, will add value to the computational epidemiology community.
| S-EPMC5433189 | biostudies-literature
Project description:Most of the existing research in assembly pathway prediction/analysis of viral capsids makes the simplifying assumption that the configuration of the intermediate states can be extracted directly from the final configuration of the entire capsid. This assumption does not take into account the conformational changes of the constituent proteins as well as minor changes to the binding interfaces that continue throughout the assembly process until stabilization. This article presents a statistical-ensemble-based approach that samples the configurational space for each monomer with the relative local orientation between monomers, to capture the uncertainties in binding and conformations. Further, instead of using larger capsomers (trimers, pentamers) as building blocks, we allow all possible subassemblies to bind in all possible combinations. We represent the resulting assembly graph in two different ways: First, we use the Wilcoxon signed-rank measure to compare the distributions of binding free energy computed on the sampled conformations to predict likely pathways. Second, we represent chemical equilibrium aspects of the transitions as a Bayesian Factor graph where both associations and dissociations are modeled based on concentrations and the binding free energies. We applied these protocols on the feline panleukopenia virus and the Nudaurelia capensis virus. Results from these experiments showed a significant departure from those that one would obtain if only the static configurations of the proteins were considered. Hence, we establish the importance of an uncertainty-aware protocol for pathway analysis, and we provide a statistical framework as an important first step toward assembly pathway prediction with high statistical confidence.
| S-EPMC5757074 | biostudies-literature