Dataset Information

Calibration methods used in cancer simulation models and suggested reporting guidelines.

ABSTRACT: Increasingly, computer simulation models are used for economic and policy evaluation in cancer prevention and control. A model's predictions of key outcomes, such as screening effectiveness, depend on the values of unobservable natural history parameters. Calibration is the process of determining the values of unobservable parameters by constraining model output to replicate observed data. Because there are many approaches for model calibration and little consensus on best practices, we surveyed the literature to catalogue the use and reporting of these methods in cancer simulation models. We conducted a MEDLINE search (1980 through 2006) for articles on cancer-screening models and supplemented search results with articles from our personal reference databases. For each article, two authors independently abstracted pre-determined items using a standard form. Data items included cancer site, model type, methods used for determination of unobservable parameter values and description of any calibration protocol. All authors reached consensus on items of disagreement. Reviews and non-cancer models were excluded. Articles describing analytical models, which estimate parameters with statistical approaches (e.g. maximum likelihood) were catalogued separately. Models that included unobservable parameters were analysed and classified by whether calibration methods were reported and if so, the methods used. The review process yielded 154 articles that met our inclusion criteria and, of these, we concluded that 131 may have used calibration methods to determine model parameters. Although the term 'calibration' was not always used, descriptions of calibration or 'model fitting' were found in 50% (n = 66) of the articles, with an additional 16% (n = 21) providing a reference to methods. Calibration target data were identified in nearly all of these articles. Other methodological details, such as the goodness-of-fit metric, were discussed in 54% (n = 47 of 87) of the articles reporting calibration methods, while few details were provided on the algorithms used to search the parameter space. Our review shows that the use of cancer simulation modelling is increasing, although thorough descriptions of calibration procedures are rare in the published literature for these models. Calibration is a key component of model development and is central to the validity and credibility of subsequent analyses and inferences drawn from model predictions. To aid peer-review and facilitate discussion of modelling methods, we propose a standardized Calibration Reporting Checklist for model documentation.

SUBMITTER: Stout NK

PROVIDER: S-EPMC2787446 | biostudies-literature | 2009

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Calibration methods used in cancer simulation models and suggested reporting guidelines.

Stout Natasha K NK Knudsen Amy B AB Kong Chung Yin CY McMahon Pamela M PM Gazelle G Scott GS

PharmacoEconomics 20090101 7

Increasingly, computer simulation models are used for economic and policy evaluation in cancer prevention and control. A model's predictions of key outcomes, such as screening effectiveness, depend on the values of unobservable natural history parameters. Calibration is the process of determining the values of unobservable parameters by constraining model output to replicate observed data. Because there are many approaches for model calibration and little consensus on best practices, we surveyed ...[more]

PMID: 19663525

Similar Datasets

Project description:ImportanceVarious model reporting guidelines have been proposed to ensure clinical prediction models are reliable and fair. However, no consensus exists about which model details are essential to report, and commonalities and differences among reporting guidelines have not been characterized. Furthermore, how well documentation of deployed models adheres to these guidelines has not been studied.ObjectivesTo assess information requested by model reporting guidelines and whether the documentation for commonly used machine learning models developed by a single vendor provides the information requested.Evidence reviewMEDLINE was queried using machine learning model card and reporting machine learning from November 4 to December 6, 2020. References were reviewed to find additional publications, and publications without specific reporting recommendations were excluded. Similar elements requested for reporting were merged into representative items. Four independent reviewers and 1 adjudicator assessed how often documentation for the most commonly used models developed by a single vendor reported the items.FindingsFrom 15 model reporting guidelines, 220 unique items were identified that represented the collective reporting requirements. Although 12 items were commonly requested (requested by 10 or more guidelines), 77 items were requested by just 1 guideline. Documentation for 12 commonly used models from a single vendor reported a median of 39% (IQR, 37%-43%; range, 31%-47%) of items from the collective reporting requirements. Many of the commonly requested items had 100% reporting rates, including items concerning outcome definition, area under the receiver operating characteristics curve, internal validation, and intended clinical use. Several items reported half the time or less related to reliability, such as external validation, uncertainty measures, and strategy for handling missing data. Other frequently unreported items related to fairness (summary statistics and subgroup analyses, including for race and ethnicity or sex).Conclusions and relevanceThese findings suggest that consistent reporting recommendations for clinical predictive models are needed for model developers to share necessary information for model deployment. The many published guidelines would, collectively, require reporting more than 200 items. Model documentation from 1 vendor reported the most commonly requested items from model reporting guidelines. However, areas for improvement were identified in reporting items related to model reliability and fairness. This analysis led to feedback to the vendor, which motivated updates to the documentation for future users.

Project description:BackgroundDevelopment of prognostic models enables identification of variables that are influential in predicting patient outcome and the use of these multiple risk factors in a systematic, reproducible way according to evidence based methods. The reliability of models depends on informed use of statistical methods, in combination with prior knowledge of disease. We reviewed published articles to assess reporting and methods used to develop new prognostic models in cancer.MethodsWe developed a systematic search string and identified articles from PubMed. Forty-seven articles were included that satisfied the following inclusion criteria: published in 2005; aiming to predict patient outcome; presenting new prognostic models in cancer with outcome time to an event and including a combination of at least two separate variables; and analysing data using multivariable analysis suitable for time to event data.ResultsIn 47 studies, prospective cohort or randomised controlled trial data were used for model development in only 33% (15) of studies. In 30% (14) of the studies insufficient data were available, having fewer than 10 events per variable (EPV) used in model development. EPV could not be calculated in a further 40% (19) of the studies. The coding of candidate variables was only reported in 68% (32) of the studies. Although use of continuous variables was reported in all studies, only one article reported using recommended methods of retaining all these variables as continuous without categorisation. Statistical methods for selection of variables in the multivariate modelling were often flawed. A method that is not recommended, namely, using statistical significance in univariate analysis as a pre-screening test to select variables for inclusion in the multivariate model, was applied in 48% (21) of the studies.ConclusionsWe found that published prognostic models are often characterised by both use of inappropriate methods for development of multivariable models and poor reporting. In addition, models are limited by the lack of studies based on prospective data of sufficient sample size to avoid overfitting. The use of poor methods compromises the reliability of prognostic models developed to provide objective probability estimates to complement clinical intuition of the physician and guidelines.

Project description:Background Improved methods for sampling outdoor-biting mosquitoes are urgently needed to improve surveillance of vector-borne diseases. Such tools could potentially replace the human landing catch (HLC), which, despite being the most direct option for measuring human exposures, raises significant ethical and logistical concerns. Several alternatives are under development, but detailed evaluation still requires common frameworks for calibration relative to HLC. The aim of this study was to develop and validate a statistical framework for predicting human-biting rates from different exposure-free alternatives. Methods We obtained mosquito abundance data (Anopheles arabiensis, Anopheles funestus and Culex spp.) from a year-long Tanzanian study comparing six outdoor traps [Suna Trap (SUN), BG Sentinel (BGS), M-Trap (MTR), M-Trap + CDC (MTRC), Ifakara Tent Trap-C (ITT-C) and Mosquito Magnet-X Trap (MMX)] and HLC. Generalised linear models were developed within a Bayesian framework to investigate associations between the traps and HLC, taking intra- and inter-specific density dependence into account. The best model was used to create a calibration tool for predicting HLC-equivalents. Results For An. arabiensis, SUN catches had the strongest correlation with HLC (R2 = 19.4), followed by BGS (R2 = 17.2) and MTRC (R2 = 13.1) catches. The least correlated catch was MMX (R2 = 2.5). For An. funestus, BGS had the strongest correlation with the HLC (R2 = 53.4), followed by MTRC (R2 = 37.4) and MTR (R2 = 37.4). For Culex mosquitoes, the traps most highly correlated with the HLC were MTR (R2 = 45.4) and MTRC (R2 = 44.2). Density dependence, both between and within species, influenced the performance of only BGS traps. An interactive Shiny App calibration tool was developed for this and similar applications. Conclusion We successfully developed a calibration tool to assess the performance of different traps for assessing outdoor-biting risk, and established a valuable framework for estimating human exposures based on the trap catches. The performance of candidate traps varied between mosquito taxa; thus, there was no single optimum. Although all the traps tested underestimated the HLC-derived exposures, it was possible to mathematically define their representativeness of the true biting risk, with or without density dependence. The results of this study emphasise the need to aim for a consistent and representative sampling approach, as opposed to simply seeking traps that catch the most mosquitoes. Graphical Abstract Supplementary Information The online version contains supplementary material available at 10.1186/s13071-022-05403-7.

Project description:BackgroundIndividual-based modeling is a growing technique in the HIV transmission and prevention literature, but insufficient attention has been paid to formally evaluate the quality of reporting in this field. We present reporting recommendations for individual-based models for HIV treatment and prevention, assess the quality of reporting in the existing literature, and comment on the contribution of this model type to HIV policy and prediction.MethodsWe developed reporting recommendations for individual-based HIV transmission mathematical models, and through a systematic search, used them to evaluate the reporting in the existing literature. We identified papers that employed individual-based simulation models and were published in English prior to December 31, 2012. Articles were included if the models they employed simulated and tracked individuals, simulated HIV transmission between individuals in a particular population, and considered a particular treatment or prevention intervention. The papers were assessed with the reporting recommendations.FindingsOf 214 full text articles examined, 32 were included in the evaluation, representing 20 independent individual-based HIV treatment and prevention mathematical models. Manuscripts universally reported the objectives, context, and modeling conclusions in the context of the modeling assumptions and the model's predictive capabilities, but the reporting of individual-based modeling methods, parameterization and calibration was variable. Six papers discussed the time step used and one discussed efforts to maintain internal validity in coding.ConclusionIndividual-based models represent detailed HIV transmission processes with the potential to contribute to inference and policy making for many different regions and populations. The rigor in reporting of assumptions, methods, and calibration of individual-based models focused on HIV transmission and prevention varies greatly. Higher standards for reporting of statistically rigorous calibration and model assumption testing need to be implemented to increase confidence in existing and future modeling results.

Project description:BackgroundCancer screening should be recommended only when the balance between benefits and harms is favorable. This review evaluated how U.S. cancer screening guidelines reported harms, within and across organ-specific processes to screen for cancer.ObjectiveTo describe current reporting practices and identify opportunities for improvement.DesignReview of guidelines.SettingUnited States.PatientsPatients eligible for screening for breast, cervical, colorectal, lung, or prostate cancer according to U.S. guidelines.MeasurementsInformation was abstracted on reporting of patient-level harms associated with screening, diagnostic follow-up, and treatment. The authors classified harms reporting as not mentioned, conceptual, qualitative, or quantitative and noted whether literature was cited when harms were described. Frequency of harms reporting was summarized by organ type.ResultsHarms reporting was inconsistent across organ types and at each step of the cancer screening process. Guidelines did not report all harms for any specific organ type or for any category of harm across organ types. The most complete harms reporting was for prostate cancer screening guidelines and the least complete for colorectal cancer screening guidelines. Conceptualization of harms and use of quantitative evidence also differed by organ type.LimitationsThis review considers only patient-level harms. The authors did not verify accuracy of harms information presented in the guidelines.ConclusionThe review identified opportunities for improving conceptualization, assessment, and reporting of screening process-related harms in guidelines. Future work should consider nuances associated with each organ-specific process to screen for cancer, including which harms are most salient and where evidence gaps exist, and explicitly explore how to optimally weigh available evidence in determining net screening benefit. Improved harms reporting could aid informed decision making, ultimately improving cancer screening delivery.Primary funding sourceNational Cancer Institute.

Dataset Information

Calibration methods used in cancer simulation models and suggested reporting guidelines.

Publications

Calibration methods used in cancer simulation models and suggested reporting guidelines.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets