Dataset Information

Adaptive Validation Design: A Bayesian Approach to Validation Substudy Design With Prospective Data Collection.

ABSTRACT: An internal validation substudy compares an imperfect measurement of a variable with a gold-standard measurement in a subset of the study population. Validation data permit calculation of a bias-adjusted estimate, which has the same expected value as the association that would have been observed had the gold-standard measurement been available for the entire study population. Existing guidance on optimal sampling for validation substudies assumes complete enrollment and follow-up of the target cohort. No guidance exists for validation substudy design while cohort data are actively being collected. In this article, we use the framework of Bayesian monitoring methods to develop an adaptive approach to validation study design. This method monitors whether sufficient validation data have been collected to meet predefined criteria for estimation of the positive and negative predictive values. We demonstrate the utility of this method using the Study of Transition, Outcomes and Gender-a cohort study of transgender and gender nonconforming people. We demonstrate the method's ability to determine efficacy (when sufficient validation data have accumulated to obtain estimates of the predictive values that fall above a threshold value) and futility (when sufficient validation data have accumulated to conclude the mismeasured variable is an untenable substitute for the gold-standard measurement). This proposed method can be applied within the context of any parent epidemiologic study design and modified to meet alternative criteria given specific study or validation study objectives. Our method provides a novel approach to effective and efficient estimation of classification parameters as validation data accrue.

SUBMITTER: Collin LJ

PROVIDER: S-EPMC7269021 | biostudies-literature | 2020 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Adaptive Validation Design: A Bayesian Approach to Validation Substudy Design With Prospective Data Collection.

Collin Lindsay J LJ MacLehose Richard F RF Ahern Thomas P TP Nash Rebecca R Getahun Darios D Roblin Douglas D Silverberg Michael J MJ Goodman Michael M Lash Timothy L TL

Epidemiology (Cambridge, Mass.) 20200701 4

An internal validation substudy compares an imperfect measurement of a variable with a gold-standard measurement in a subset of the study population. Validation data permit calculation of a bias-adjusted estimate, which has the same expected value as the association that would have been observed had the gold-standard measurement been available for the entire study population. Existing guidance on optimal sampling for validation substudies assumes complete enrollment and follow-up of the target c ...[more]

PMID: 32483065

Similar Datasets

Project description:BackgroundAmong men and women diagnosed with colorectal cancer (CRC), 20-50% will develop a cancer recurrence. Cancer recurrences are not routinely captured by most population-based registries; however, linkage across Danish registries allows for the development of predictive models to detect recurrence. Successful application of such models in population-based settings requires validation against a gold standard to ensure the accuracy of recurrence identification.ObjectiveWe apply a recently developed validation study design for prospectively collected validation data to validate predicted CRC recurrences against gold standard diagnoses from medical records in an actively followed cohort of CRC patients in Denmark.MethodsWe use a Bayesian monitoring framework, traditionally used in clinical trials, to iteratively update classification parameters (positive and negative predictive values, and sensitivity and specificity) in an adaptive validation substudy design. This design allows determination of the sample size necessary to estimate the corresponding parameters and to identify when validation efforts can cease based on predefined criteria for parameter values and levels of precision.ResultsAmong 355 men and women diagnosed with CRC in Denmark and actively followed semi-annually, there were 63 recurrences diagnosed by active follow-up and 70 recurrences identified by a predictive algorithm. The adaptive validation design met stopping criteria for the classification parameters after 120 patients had their recurrence information validated. This stopping point yielded parameter estimates for the classification parameters similar to those obtained when the entire cohort was validated, with 66% less patients needed for the validation study.ConclusionIn this proof of concept application of the adaptive validation study design for outcome misclassification, we demonstrated the ability of the method to accurately determine when sufficient validation data have been collected. This method serves as a novel validation substudy design for prospectively collected data with simultaneous implementation of a validation study.

Project description:Developing data-driven solutions that address real-world problems requires understanding of these problems' causes and how their interaction affects the outcome-often with only observational data. Causal Bayesian Networks (BN) have been proposed as a powerful method for discovering and representing the causal relationships from observational data as a Directed Acyclic Graph (DAG). BNs could be especially useful for research in global health in Lower and Middle Income Countries, where there is an increasing abundance of observational data that could be harnessed for policy making, program evaluation, and intervention design. However, BNs have not been widely adopted by global health professionals, and in real-world applications, confidence in the results of BNs generally remains inadequate. This is partially due to the inability to validate against some ground truth, as the true DAG is not available. This is especially problematic if a learned DAG conflicts with pre-existing domain doctrine. Here we conceptualize and demonstrate an idea of a "Causal Datasheet" that could approximate and document BN performance expectations for a given dataset, aiming to provide confidence and sample size requirements to practitioners. To generate results for such a Causal Datasheet, a tool was developed which can generate synthetic Bayesian networks and their associated synthetic datasets to mimic real-world datasets. The results given by well-known structure learning algorithms and a novel implementation of the OrderMCMC method using the Quotient Normalized Maximum Likelihood score were recorded. These results were used to populate the Causal Datasheet, and recommendations could be made dependent on whether expected performance met user-defined thresholds. We present our experience in the creation of Causal Datasheets to aid analysis decisions at different stages of the research process. First, one was deployed to help determine the appropriate sample size of a planned study of sexual and reproductive health in Madhya Pradesh, India. Second, a datasheet was created to estimate the performance of an existing maternal health survey we conducted in Uttar Pradesh, India. Third, we validated generated performance estimates and investigated current limitations on the well-known ALARM dataset. Our experience demonstrates the utility of the Causal Datasheet, which can help global health practitioners gain more confidence when applying BNs.

Dataset Information

Adaptive Validation Design: A Bayesian Approach to Validation Substudy Design With Prospective Data Collection.

Publications

Adaptive Validation Design: A Bayesian Approach to Validation Substudy Design With Prospective Data Collection.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets