Dataset Information

Evaluation of diagnostic accuracy in detecting ordered symptom statuses without a gold standard.

ABSTRACT: Our research is motivated by 2 methodological problems in assessing diagnostic accuracy of traditional Chinese medicine (TCM) doctors in detecting a particular symptom whose true status has an ordinal scale and is unknown-imperfect gold standard bias and ordinal scale symptom status. In this paper, we proposed a nonparametric maximum likelihood method for estimating and comparing the accuracy of different doctors in detecting a particular symptom without a gold standard when the true symptom status had an ordered multiple class. In addition, we extended the concept of the area under the receiver operating characteristic curve to a hyper-dimensional overall accuracy for diagnostic accuracy and alternative graphs for displaying a visual result. The simulation studies showed that the proposed method had good performance in terms of bias and mean squared error. Finally, we applied our method to our motivating example on assessing the diagnostic abilities of 5 TCM doctors in detecting symptoms related to Chills disease.

SUBMITTER: Wang Z

PROVIDER: S-EPMC3114651 | biostudies-literature | 2011 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluation of diagnostic accuracy in detecting ordered symptom statuses without a gold standard.

Wang Zheyu Z Zhou Xiao-Hua XH Wang Miqu M

Biostatistics (Oxford, England) 20110105 3

Our research is motivated by 2 methodological problems in assessing diagnostic accuracy of traditional Chinese medicine (TCM) doctors in detecting a particular symptom whose true status has an ordinal scale and is unknown-imperfect gold standard bias and ordinal scale symptom status. In this paper, we proposed a nonparametric maximum likelihood method for estimating and comparing the accuracy of different doctors in detecting a particular symptom without a gold standard when the true symptom sta ...[more]

PMID: 21209155

Similar Datasets

Project description:ObjectiveThe Bayesian model plays an important role in diagnostic test evaluation in the absence of the gold standard, which used the external prior distribution of a parameter combined with sample data to yield the posterior distribution of the test characteristics. However, the correlation between diagnostic tests has always been a problem that cannot be ignored in the Bayesian model evaluation. This study will discuss how different Bayesian model, correlation scenarios, and prior distribution affect the outcome.MethodsThe data analyzed in this study was gathered during studies of patients presenting to the Nanjing Chest Hospital with suspected tuberculosis. The diagnostic character of T-SPOT.Tb and KD38 tuberculosis antibody test were evaluated in different Bayesian model, and discharge diagnosis as a gold standard was used to verify the model results in the end.ResultThe comparison of four models under the conditional independence situation found that Bayesian probabilistic constraint model was consistent with the Conditional Covariance Bayesian model. The results were mainly affected by prior information. The sensitivity and specificity of the two tests in Conditional Covariance Bayesian model in prior constraint situation were considerably higher than the Bayesian probabilistic constraint model in prior constraint situation. The results of the four models under the conditional dependence situation were similar to the conditional independence situation; pD was also negative with no prior constraint situation in both model Bayesian probabilistic constraint model and Conditional Covariance Bayesian model. The Deviance Information Criterion of Bayesian probabilistic constraint model was close to model Conditional Covariance Bayesian model, but pD of Conditional Covariance Bayesian model in Prior constraint situation (pD=2.40) was higher than the Bayesian probabilistic constraint model in Prior constraint situation (pD=1.66).ConclusionThe result of Conditional Covariance Bayesian model in prior constraint with conditional independence situation was closest to the result of gold standard evaluation in our data. Both of the two Bayesian methods are the feasible way for the evaluation of diagnostic test in the absence of the gold standard diagnostic. Prior source, priority number, and conditional dependencies should be considered in the method selection, the accuracy of posterior estimation mainly depending on the prior distribution.

Project description:ObjectiveTo systematically review methods developed and employed to evaluate the diagnostic accuracy of medical test when there is a missing or no gold standard.Study design and settingsArticles that proposed or applied any methods to evaluate the diagnostic accuracy of medical test(s) in the absence of gold standard were reviewed. The protocol for this review was registered in PROSPERO (CRD42018089349).ResultsIdentified methods were classified into four main groups: methods employed when there is a missing gold standard; correction methods (which make adjustment for an imperfect reference standard with known diagnostic accuracy measures); methods employed to evaluate a medical test using multiple imperfect reference standards; and other methods, like agreement studies, and a mixed group of alternative study designs. Fifty-one statistical methods were identified from the review that were developed to evaluate medical test(s) when the true disease status of some participants is unverified with the gold standard. Seven correction methods were identified and four methods were identified to evaluate medical test(s) using multiple imperfect reference standards. Flow-diagrams were developed to guide the selection of appropriate methods.ConclusionVarious methods have been proposed to evaluate medical test(s) in the absence of a gold standard for some or all participants in a diagnostic accuracy study. These methods depend on the availability of the gold standard, its' application to the participants in the study and the availability of alternative reference standard(s). The clinical application of some of these methods, especially methods developed when there is missing gold standard is however limited. This may be due to the complexity of these methods and/or a disconnection between the fields of expertise of those who develop (e.g. mathematicians) and those who employ the methods (e.g. clinical researchers). This review aims to help close this gap with our classification and guidance tools.

Project description:BackgroundA large number of web-based COVID-19 symptom checkers and chatbots have been developed; however, anecdotal evidence suggests that their conclusions are highly variable. To our knowledge, no study has evaluated the accuracy of COVID-19 symptom checkers in a statistically rigorous manner.ObjectiveThe aim of this study is to evaluate and compare the diagnostic accuracies of web-based COVID-19 symptom checkers.MethodsWe identified 10 web-based COVID-19 symptom checkers, all of which were included in the study. We evaluated the COVID-19 symptom checkers by assessing 50 COVID-19 case reports alongside 410 non-COVID-19 control cases. A bootstrapping method was used to counter the unbalanced sample sizes and obtain confidence intervals (CIs). Results are reported as sensitivity, specificity, F1 score, and Matthews correlation coefficient (MCC).ResultsThe classification task between COVID-19-positive and COVID-19-negative for "high risk" cases among the 460 test cases yielded (sorted by F1 score): Symptoma (F1=0.92, MCC=0.85), Infermedica (F1=0.80, MCC=0.61), US Centers for Disease Control and Prevention (CDC) (F1=0.71, MCC=0.30), Babylon (F1=0.70, MCC=0.29), Cleveland Clinic (F1=0.40, MCC=0.07), Providence (F1=0.40, MCC=0.05), Apple (F1=0.29, MCC=-0.10), Docyet (F1=0.27, MCC=0.29), Ada (F1=0.24, MCC=0.27) and Your.MD (F1=0.24, MCC=0.27). For "high risk" and "medium risk" combined the performance was: Symptoma (F1=0.91, MCC=0.83) Infermedica (F1=0.80, MCC=0.61), Cleveland Clinic (F1=0.76, MCC=0.47), Providence (F1=0.75, MCC=0.45), Your.MD (F1=0.72, MCC=0.33), CDC (F1=0.71, MCC=0.30), Babylon (F1=0.70, MCC=0.29), Apple (F1=0.70, MCC=0.25), Ada (F1=0.42, MCC=0.03), and Docyet (F1=0.27, MCC=0.29).ConclusionsWe found that the number of correctly assessed COVID-19 and control cases varies considerably between symptom checkers, with different symptom checkers showing different strengths with respect to sensitivity and specificity. A good balance between sensitivity and specificity was only achieved by two symptom checkers.

Dataset Information

Evaluation of diagnostic accuracy in detecting ordered symptom statuses without a gold standard.

Publications

Evaluation of diagnostic accuracy in detecting ordered symptom statuses without a gold standard.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets