Dataset Information

A Bayesian framework for estimating the incremental value of a diagnostic test in the absence of a gold standard.

ABSTRACT:

Background

The absence of a gold standard, i.e., a diagnostic reference standard having perfect sensitivity and specificity, is a common problem in clinical practice and in diagnostic research studies. There is a need for methods to estimate the incremental value of a new, imperfect test in this context.

Methods

We use a Bayesian approach to estimate the probability of the unknown disease status via a latent class model and extend two commonly-used measures of incremental value based on predictive values [difference in the area under the ROC curve (AUC) and integrated discrimination improvement (IDI)] to the context where no gold standard exists. The methods are illustrated using simulated data and applied to the problem of estimating the incremental value of a novel interferon-gamma release assay (IGRA) over the tuberculin skin test (TST) for latent tuberculosis (TB) screening. We also show how to estimate the incremental value of IGRAs when decisions are based on observed test results rather than predictive values.

Results

We showed that the incremental value is greatest when both sensitivity and specificity of the new test are better and that conditional dependence between the tests reduces the incremental value. The incremental value of the IGRA depends on the sensitivity and specificity of the TST, as well as the prevalence of latent TB, and may thus vary in different populations.

Conclusions

Even in the absence of a gold standard, incremental value statistics may be estimated and can aid decisions about the practical value of a new diagnostic test.

SUBMITTER: Ling DI

PROVIDER: S-EPMC4077291 | biostudies-literature | 2014 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A Bayesian framework for estimating the incremental value of a diagnostic test in the absence of a gold standard.

Ling Daphne I DI Pai Madhukar M Schiller Ian I Dendukuri Nandini N

BMC medical research methodology 20140515

<h4>Background</h4>The absence of a gold standard, i.e., a diagnostic reference standard having perfect sensitivity and specificity, is a common problem in clinical practice and in diagnostic research studies. There is a need for methods to estimate the incremental value of a new, imperfect test in this context.<h4>Methods</h4>We use a Bayesian approach to estimate the probability of the unknown disease status via a latent class model and extend two commonly-used measures of incremental value ba ...[more]

PMID: 24886359

Similar Datasets

Project description:ObjectiveThe Bayesian model plays an important role in diagnostic test evaluation in the absence of the gold standard, which used the external prior distribution of a parameter combined with sample data to yield the posterior distribution of the test characteristics. However, the correlation between diagnostic tests has always been a problem that cannot be ignored in the Bayesian model evaluation. This study will discuss how different Bayesian model, correlation scenarios, and prior distribution affect the outcome.MethodsThe data analyzed in this study was gathered during studies of patients presenting to the Nanjing Chest Hospital with suspected tuberculosis. The diagnostic character of T-SPOT.Tb and KD38 tuberculosis antibody test were evaluated in different Bayesian model, and discharge diagnosis as a gold standard was used to verify the model results in the end.ResultThe comparison of four models under the conditional independence situation found that Bayesian probabilistic constraint model was consistent with the Conditional Covariance Bayesian model. The results were mainly affected by prior information. The sensitivity and specificity of the two tests in Conditional Covariance Bayesian model in prior constraint situation were considerably higher than the Bayesian probabilistic constraint model in prior constraint situation. The results of the four models under the conditional dependence situation were similar to the conditional independence situation; pD was also negative with no prior constraint situation in both model Bayesian probabilistic constraint model and Conditional Covariance Bayesian model. The Deviance Information Criterion of Bayesian probabilistic constraint model was close to model Conditional Covariance Bayesian model, but pD of Conditional Covariance Bayesian model in Prior constraint situation (pD=2.40) was higher than the Bayesian probabilistic constraint model in Prior constraint situation (pD=1.66).ConclusionThe result of Conditional Covariance Bayesian model in prior constraint with conditional independence situation was closest to the result of gold standard evaluation in our data. Both of the two Bayesian methods are the feasible way for the evaluation of diagnostic test in the absence of the gold standard diagnostic. Prior source, priority number, and conditional dependencies should be considered in the method selection, the accuracy of posterior estimation mainly depending on the prior distribution.

Project description:BackgroundSpatial modeling is increasingly utilized to elucidate relationships between demographic, environmental, and socioeconomic factors, and infectious disease prevalence data. However, there is a paucity of studies focusing on spatio-temporal modeling that take into account the uncertainty of diagnostic techniques.Methodology/principal findingsWe obtained Schistosoma japonicum prevalence data, based on a standardized indirect hemagglutination assay (IHA), from annual reports from 114 schistosome-endemic villages in Dangtu County, southeastern part of the People's Republic of China, for the period 1995 to 2004. Environmental data were extracted from satellite images. Socioeconomic data were available from village registries. We used Bayesian spatio-temporal models, accounting for the sensitivity and specificity of the IHA test via an equation derived from the law of total probability, to relate the observed with the 'true' prevalence. The risk of S. japonicum was positively associated with the mean land surface temperature, and negatively correlated with the mean normalized difference vegetation index and distance to the nearest water body. There was no significant association between S. japonicum and socioeconomic status of the villages surveyed. The spatial correlation structures of the observed S. japonicum seroprevalence and the estimated infection prevalence differed from one year to another. Variance estimates based on a model adjusted for the diagnostic error were larger than unadjusted models. The generated prediction map for 2005 showed that most of the former and current infections occur in close proximity to the Yangtze River.Conclusion/significanceBayesian spatial-temporal modeling incorporating diagnostic uncertainty is a suitable approach for risk mapping S. japonicum prevalence data. The Yangtze River and its tributaries govern schistosomiasis transmission in Dangtu County, but spatial correlation needs to be taken into consideration when making risk prediction at small scales.

Project description:ObjectiveTo systematically review methods developed and employed to evaluate the diagnostic accuracy of medical test when there is a missing or no gold standard.Study design and settingsArticles that proposed or applied any methods to evaluate the diagnostic accuracy of medical test(s) in the absence of gold standard were reviewed. The protocol for this review was registered in PROSPERO (CRD42018089349).ResultsIdentified methods were classified into four main groups: methods employed when there is a missing gold standard; correction methods (which make adjustment for an imperfect reference standard with known diagnostic accuracy measures); methods employed to evaluate a medical test using multiple imperfect reference standards; and other methods, like agreement studies, and a mixed group of alternative study designs. Fifty-one statistical methods were identified from the review that were developed to evaluate medical test(s) when the true disease status of some participants is unverified with the gold standard. Seven correction methods were identified and four methods were identified to evaluate medical test(s) using multiple imperfect reference standards. Flow-diagrams were developed to guide the selection of appropriate methods.ConclusionVarious methods have been proposed to evaluate medical test(s) in the absence of a gold standard for some or all participants in a diagnostic accuracy study. These methods depend on the availability of the gold standard, its' application to the participants in the study and the availability of alternative reference standard(s). The clinical application of some of these methods, especially methods developed when there is missing gold standard is however limited. This may be due to the complexity of these methods and/or a disconnection between the fields of expertise of those who develop (e.g. mathematicians) and those who employ the methods (e.g. clinical researchers). This review aims to help close this gap with our classification and guidance tools.

Project description:BackgroundMany problems in bioinformatics involve classification based on features such as sequence, structure or morphology. Given multiple classifiers, two crucial questions arise: how does their performance compare, and how can they best be combined to produce a better classifier? A classifier can be evaluated in terms of sensitivity and specificity using benchmark, or gold standard, data, that is, data for which the true classification is known. However, a gold standard is not always available. Here we demonstrate that a Bayesian model for comparing medical diagnostics without a gold standard can be successfully applied in the bioinformatics domain, to genomic scale data sets. We present a new implementation, which unlike previous implementations is applicable to any number of classifiers. We apply this model, for the first time, to the problem of finding the globally optimal logical combination of classifiers.ResultsWe compared three classifiers of protein subcellular localisation, and evaluated our estimates of sensitivity and specificity against estimates obtained using a gold standard. The method overestimated sensitivity and specificity with only a small discrepancy, and correctly ranked the classifiers. Diagnostic tests for swine flu were then compared on a small data set. Lastly, classifiers for a genome-wide association study of macular degeneration with 541094 SNPs were analysed. In all cases, run times were feasible, and results precise. The optimal logical combination of classifiers was also determined for all three data sets. Code and data are available from http://bioinformatics.monash.edu.au/downloads/.ConclusionsThe examples demonstrate the methods are suitable for both small and large data sets, applicable to the wide range of bioinformatics classification problems, and robust to dependence between classifiers. In all three test cases, the globally optimal logical combination of the classifiers was found to be their union, according to three out of four ranking criteria. We propose as a general rule of thumb that the union of classifiers will be close to optimal.

Dataset Information

A Bayesian framework for estimating the incremental value of a diagnostic test in the absence of a gold standard.

Background

Methods

Results

Conclusions

Publications

A Bayesian framework for estimating the incremental value of a diagnostic test in the absence of a gold standard.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets