Dataset Information

Validation of population-based disease simulation models: a review of concepts and methods.

ABSTRACT:

Background

Computer simulation models are used increasingly to support public health research and policy, but questions about their quality persist. The purpose of this article is to review the principles and methods for validation of population-based disease simulation models.

Methods

We developed a comprehensive framework for validating population-based chronic disease simulation models and used this framework in a review of published model validation guidelines. Based on the review, we formulated a set of recommendations for gathering evidence of model credibility.

Results

Evidence of model credibility derives from examining: 1) the process of model development, 2) the performance of a model, and 3) the quality of decisions based on the model. Many important issues in model validation are insufficiently addressed by current guidelines. These issues include a detailed evaluation of different data sources, graphical representation of models, computer programming, model calibration, between-model comparisons, sensitivity analysis, and predictive validity. The role of external data in model validation depends on the purpose of the model (e.g., decision analysis versus prediction). More research is needed on the methods of comparing the quality of decisions based on different models.

Conclusion

As the role of simulation modeling in population health is increasing and models are becoming more complex, there is a need for further improvements in model validation methodology and common standards for evaluating model credibility.

SUBMITTER: Kopec JA

PROVIDER: S-EPMC3001435 | biostudies-literature | 2010 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Validation of population-based disease simulation models: a review of concepts and methods.

Kopec Jacek A JA Finès Philippe P Manuel Douglas G DG Buckeridge David L DL Flanagan William M WM Oderkirk Jillian J Abrahamowicz Michal M Harper Samuel S Sharif Behnam B Okhmatovskaia Anya A Sayre Eric C EC Rahman M Mushfiqur MM Wolfson Michael C MC

BMC public health 20101118

<h4>Background</h4>Computer simulation models are used increasingly to support public health research and policy, but questions about their quality persist. The purpose of this article is to review the principles and methods for validation of population-based disease simulation models.<h4>Methods</h4>We developed a comprehensive framework for validating population-based chronic disease simulation models and used this framework in a review of published model validation guidelines. Based on the re ...[more]

PMID: 21087466

Similar Datasets

Project description:Detailed modeling and simulation of biochemical systems is complicated by the problem of combinatorial complexity, an explosion in the number of species and reactions due to myriad protein-protein interactions and post-translational modifications. Rule-based modeling overcomes this problem by representing molecules as structured objects and encoding their interactions as pattern-based rules. This greatly simplifies the process of model specification, avoiding the tedious and error prone task of manually enumerating all species and reactions that can potentially exist in a system. From a simulation perspective, rule-based models can be expanded algorithmically into fully-enumerated reaction networks and simulated using a variety of network-based simulation methods, such as ordinary differential equations or Gillespie's algorithm, provided that the network is not exceedingly large. Alternatively, rule-based models can be simulated directly using particle-based kinetic Monte Carlo methods. This "network-free" approach produces exact stochastic trajectories with a computational cost that is independent of network size. However, memory and run time costs increase with the number of particles, limiting the size of system that can be feasibly simulated. Here, we present a hybrid particle/population simulation method that combines the best attributes of both the network-based and network-free approaches. The method takes as input a rule-based model and a user-specified subset of species to treat as population variables rather than as particles. The model is then transformed by a process of "partial network expansion" into a dynamically equivalent form that can be simulated using a population-adapted network-free simulator. The transformation method has been implemented within the open-source rule-based modeling platform BioNetGen, and resulting hybrid models can be simulated using the particle-based simulator NFsim. Performance tests show that significant memory savings can be achieved using the new approach and a monetary cost analysis provides a practical measure of its utility.

Project description:Spatial capture-recapture (SCR) models have improved the ability to estimate densities of rare and elusive animals. However, SCR models have seldom been validated even as model formulations diversify and expand to incorporate new sampling methods and/or additional sources of information on model parameters. Information on the relationship between encounter probabilities, sources of additional information, and the reliability of density estimates, is rare but crucial to assessing reliability of SCR-based estimates. We used a simulation-based approach that incorporated prior empirical work to assess the accuracy and precision of density estimates from SCR models using spatially unstructured sampling. To assess the consequences of sparse data and potential sources of bias, we simulated data under six scenarios corresponding to three different levels of search effort and two levels of correlation between search effort and animal density. We then estimated density for each scenario using four models that included increasing amounts of information from harvested individuals and telemetry to evaluate the impact of additional sources of information. Model results were sensitive to the quantity of available information: density estimates based on low search effort were biased high and imprecise, whereas estimates based on high search effort were unbiased and precise. A correlation between search effort and animal density resulted in a positive bias in density estimates, though the bias decreased with increasingly informative datasets. Adding information from harvested individuals and telemetered individuals improved density estimates based on low and moderate effort but had negligible impact for datasets resulting from high effort. We demonstrated that density estimates from SCR models using spatially unstructured sampling are reliable when sufficient information is provided. Accurate density estimates can result if empirical-based simulations such as those presented here are used to develop study designs with appropriate amounts of effort and information sources.

Project description:BackgroundMicro-simulation models of risk-factors and chronic diseases are built increasingly often, and each model starts with an initial population. Constructing such populations when no survey data covering all variables are available is no trivial task, often requiring complex methods based on several (untested) assumptions. In this paper, we propose a method for evaluating the merits of construction methods, and apply this to one specific method: the construction method used in the DYNAMO-HIA model.MethodsThe initial population constructed using the DYNAMO-HIA method is compared to another population constructed by starting a simulation with only newborns and simulating the course taken by one risk-factor and several diseases. In this simulation, the age- and sex-specific prevalence of the risk-factor is kept constant over time.ResultsOur simulations show that, in general, the DYNAMO-HIA method clearly outperforms a method that assumes independence of the risk-factor and the prevalence of diseases and independence between all diseases. In many situations the DYNAMO-HIA method performs reasonably well, but in some the proportion with the risk-factor for those with a disease is under- or overestimated by as much as 10 percentage points. For determining comorbidity between diseases linked by a common causal disease or a common risk-factor it also performs reasonably well. However, the current method performs poorly for determining the comorbidity between one disease caused by the other.ConclusionThe DYNAMO-HIA methods perform reasonably well; they outperform a baseline assumption of independence between the risk-factor and diseases in the initial population. The method for determining the comorbidity between diseases that are causally linked needs improvement. Given the existing discrepancies for situations with high relative risks, however, developing more elaborate methods based on running simulation models to generate an initial population would be worthwhile.

Dataset Information

Validation of population-based disease simulation models: a review of concepts and methods.

Background

Methods

Results

Conclusion

Publications

Validation of population-based disease simulation models: a review of concepts and methods.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets