Dataset Information

Comparative diagnostic accuracy studies with an imperfect reference standard - a comparison of correction methods.

ABSTRACT:

Background

Staquet et al. and Brenner both developed correction methods to estimate the sensitivity and specificity of a binary-response index test when the reference standard is imperfect and its sensitivity and specificity are known. However, to our knowledge, no study has compared the statistical properties of these methods, despite their long application in diagnostic accuracy studies.

Aim

To compare the correction methods developed by Staquet et al. and Brenner.

Methods

Simulations techniques were employed to compare the methods under assumptions that the new test and the reference standard are conditionally independent or dependent given the true disease status of an individual. Three clinical datasets were analysed to understand the impact of using each method to inform clinical decision-making.

Results

Under the assumption of conditional independence, the Staquet et al. correction method outperforms the Brenner correction method irrespective of the prevalence of disease and whether the performance of the reference standard is better or worse than the index test. However, when the prevalence of the disease is high (> 0.9) or low (< 0.1), the Staquet et al. correction method can produce illogical results (i.e. results outside [0,1]). Under the assumption of conditional dependence; both methods failed to estimate the sensitivity and specificity of the index test especially when the covariance terms between the index test and the reference standard is not close to zero.

Conclusion

When the new test and the imperfect reference standard are conditionally independent, and the sensitivity and specificity of the imperfect reference standard are known, the Staquet et al. correction method outperforms the Brenner method. However, where the prevalence of the target condition is very high or low or the two tests are conditionally dependent, other statistical methods such as latent class approaches should be considered.

SUBMITTER: Umemneku Chikere CM

PROVIDER: S-EPMC8040223 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundWhole blood is frequently utilized in genome-wide association studies of DNA methylation patterns in relation to environmental exposures or clinical outcomes. These associations can be confounded by cellular heterogeneity. Algorithms have been developed to measure or adjust for this heterogeneity, and some have been compared in the literature. However, with new methods available, it is unknown whether the findings will be consistent, if not which method(s) perform better.ResultsMethods: We compared eight cell-type correction methods including the method in the minfi R package, the method by Houseman et al., the Removing unwanted variation (RUV) approach, the methods in FaST-LMM-EWASher, ReFACTor, RefFreeEWAS, and RefFreeCellMix R programs, along with one approach utilizing surrogate variables (SVAs). We first evaluated the association of DNA methylation at each CpG across the whole genome with prenatal arsenic exposure levels and with cancer status, adjusted for estimated cell-type information obtained from different methods. We then compared CpGs showing statistical significance from different approaches. For the methods implemented in minfi and proposed by Houseman et al., we utilized homogeneous data with composition of some blood cells available and compared them with the estimated cell compositions. Finally, for methods not explicitly estimating cell compositions, we evaluated their performance using simulated DNA methylation data with a set of latent variables representing "cell types".ResultsResults from the SVA-based method overall showed the highest agreement with all other methods except for FaST-LMM-EWASher. Using homogeneous data, minfi provided better estimations on cell types compared to the originally proposed method by Houseman et al. Further simulation studies on methods free of reference data revealed that SVA provided good sensitivities and specificities, RefFreeCellMix in general produced high sensitivities but specificities tended to be low when confounding is present, and FaST-LMM-EWASher gave the lowest sensitivity but highest specificity.ConclusionsResults from real data and simulations indicated that SVA is recommended when the focus is on the identification of informative CpGs. When appropriate reference data are available, the method implemented in the minfi package is recommended. However, if no such reference data are available or if the focus is not on estimating cell proportions, the SVA method is suggested.

Project description:ObjectiveThe Bayesian model plays an important role in diagnostic test evaluation in the absence of the gold standard, which used the external prior distribution of a parameter combined with sample data to yield the posterior distribution of the test characteristics. However, the correlation between diagnostic tests has always been a problem that cannot be ignored in the Bayesian model evaluation. This study will discuss how different Bayesian model, correlation scenarios, and prior distribution affect the outcome.MethodsThe data analyzed in this study was gathered during studies of patients presenting to the Nanjing Chest Hospital with suspected tuberculosis. The diagnostic character of T-SPOT.Tb and KD38 tuberculosis antibody test were evaluated in different Bayesian model, and discharge diagnosis as a gold standard was used to verify the model results in the end.ResultThe comparison of four models under the conditional independence situation found that Bayesian probabilistic constraint model was consistent with the Conditional Covariance Bayesian model. The results were mainly affected by prior information. The sensitivity and specificity of the two tests in Conditional Covariance Bayesian model in prior constraint situation were considerably higher than the Bayesian probabilistic constraint model in prior constraint situation. The results of the four models under the conditional dependence situation were similar to the conditional independence situation; pD was also negative with no prior constraint situation in both model Bayesian probabilistic constraint model and Conditional Covariance Bayesian model. The Deviance Information Criterion of Bayesian probabilistic constraint model was close to model Conditional Covariance Bayesian model, but pD of Conditional Covariance Bayesian model in Prior constraint situation (pD=2.40) was higher than the Bayesian probabilistic constraint model in Prior constraint situation (pD=1.66).ConclusionThe result of Conditional Covariance Bayesian model in prior constraint with conditional independence situation was closest to the result of gold standard evaluation in our data. Both of the two Bayesian methods are the feasible way for the evaluation of diagnostic test in the absence of the gold standard diagnostic. Prior source, priority number, and conditional dependencies should be considered in the method selection, the accuracy of posterior estimation mainly depending on the prior distribution.