Dataset Information

Using Bayes theorem to estimate positive and negative predictive values for continuously and ordinally scaled diagnostic tests.

ABSTRACT:

Objectives

Positive predictive values (PPVs) and negative predictive values (NPVs) are frequently reported to put estimates of accuracy of a diagnostic test in clinical context and to obtain risk estimates for a given patient taking into account baseline prevalence in the population. In order to calculate PPV and NPV, tests with ordinally or continuously scaled results are commonly dichotomized at the expense of a loss of information.

Methods

Extending the rationale for the calculation of PPV and NPV, Bayesian theorem is used to calculate the probability of disease given the outcome of a continuously or ordinally scaled test. Probabilities of test results conditional on disease status are modeled in a Bayesian framework and subsequently transformed to probabilities of disease status conditional on test result.

Results

Using publicly available data, probability of a clinical depression diagnosis given PROMIS Depression scores was estimated. Comparison with PPV and NPV based on dichotomized scores shows that a more fine-grained interpretation of test scores is possible.

Conclusions

The proposed method bears the chance to facilitate accurate and meaningful interpretation of test results in clinical settings by avoiding unnecessary dichotomization of test scores.

SUBMITTER: Fischer F

PROVIDER: S-EPMC8170576 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundMachine learning (ML) can be an effective tool to extract information from attribute-rich molecular datasets for the generation of molecular diagnostic tests. However, the way in which the resulting scores or classifications are produced from the input data may not be transparent. Algorithmic explainability or interpretability has become a focus of ML research. Shapley values, first introduced in game theory, can provide explanations of the result generated from a specific set of input data by a complex ML algorithm.MethodsFor a multivariate molecular diagnostic test in clinical use (the VeriStrat® test), we calculate and discuss the interpretation of exact Shapley values. We also employ some standard approximation techniques for Shapley value computation (local interpretable model-agnostic explanation (LIME) and Shapley Additive Explanations (SHAP) based methods) and compare the results with exact Shapley values.ResultsExact Shapley values calculated for data collected from a cohort of 256 patients showed that the relative importance of attributes for test classification varied by sample. While all eight features used in the VeriStrat® test contributed equally to classification for some samples, other samples showed more complex patterns of attribute importance for classification generation. Exact Shapley values and Shapley-based interaction metrics were able to provide interpretable classification explanations at the sample or patient level, while patient subgroups could be defined by comparing Shapley value profiles between patients. LIME and SHAP approximation approaches, even those seeking to include correlations between attributes, produced results that were quantitatively and, in some cases qualitatively, different from the exact Shapley values.ConclusionsShapley values can be used to determine the relative importance of input attributes to the result generated by a multivariate molecular diagnostic test for an individual sample or patient. Patient subgroups defined by Shapley value profiles may motivate translational research. However, correlations inherent in molecular data and the typically small ML training sets available for molecular diagnostic test development may cause some approximation methods to produce approximate Shapley values that differ both qualitatively and quantitatively from exact Shapley values. Hence, caution is advised when using approximate methods to evaluate Shapley explanations of the results of molecular diagnostic tests.