Dataset Information

Empirical assessment of bias in machine learning diagnostic test accuracy studies.

ABSTRACT:

Objective

Machine learning (ML) diagnostic tools have significant potential to improve health care. However, methodological pitfalls may affect diagnostic test accuracy studies used to appraise such tools. We aimed to evaluate the prevalence and reporting of design characteristics within the literature. Further, we sought to empirically assess whether design features may be associated with different estimates of diagnostic accuracy.

Materials and methods

We systematically retrieved 2 × 2 tables (n = 281) describing the performance of ML diagnostic tools, derived from 114 publications in 38 meta-analyses, from PubMed. Data extracted included test performance, sample sizes, and design features. A mixed-effects metaregression was run to quantify the association between design features and diagnostic accuracy.

Results

Participant ethnicity and blinding in test interpretation was unreported in 90% and 60% of studies, respectively. Reporting was occasionally lacking for rudimentary characteristics such as study design (28% unreported). Internal validation without appropriate safeguards was used in 44% of studies. Several design features were associated with larger estimates of accuracy, including having unreported (relative diagnostic odds ratio [RDOR], 2.11; 95% confidence interval [CI], 1.43-3.1) or case-control study designs (RDOR, 1.27; 95% CI, 0.97-1.66), and recruiting participants for the index test (RDOR, 1.67; 95% CI, 1.08-2.59).

Discussion

Significant underreporting of experimental details was present. Study design features may affect estimates of diagnostic performance in the ML diagnostic test accuracy literature.

Conclusions

The present study identifies pitfalls that threaten the validity, generalizability, and clinical value of ML diagnostic tools and provides recommendations for improvement.

SUBMITTER: Crowley RJ

PROVIDER: S-EPMC7647361 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:PurposeEarly clinical recognition of sepsis can be challenging. With the advancement of machine learning, promising real-time models to predict sepsis have emerged. We assessed their performance by carrying out a systematic review and meta-analysis.MethodsA systematic search was performed in PubMed, Embase.com and Scopus. Studies targeting sepsis, severe sepsis or septic shock in any hospital setting were eligible for inclusion. The index test was any supervised machine learning model for real-time prediction of these conditions. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, with a tailored Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist to evaluate risk of bias. Models with a reported area under the curve of the receiver operating characteristic (AUROC) metric were meta-analyzed to identify strongest contributors to model performance.ResultsAfter screening, a total of 28 papers were eligible for synthesis, from which 130 models were extracted. The majority of papers were developed in the intensive care unit (ICU, n = 15; 54%), followed by hospital wards (n = 7; 25%), the emergency department (ED, n = 4; 14%) and all of these settings (n = 2; 7%). For the prediction of sepsis, diagnostic test accuracy assessed by the AUROC ranged from 0.68-0.99 in the ICU, to 0.96-0.98 in-hospital and 0.87 to 0.97 in the ED. Varying sepsis definitions limit pooling of the performance across studies. Only three papers clinically implemented models with mixed results. In the multivariate analysis, temperature, lab values, and model type contributed most to model performance.ConclusionThis systematic review and meta-analysis show that on retrospective data, individual machine learning models can accurately predict sepsis onset ahead of time. Although they present alternatives to traditional scoring systems, between-study heterogeneity limits the assessment of pooled results. Systematic reporting and clinical implementation studies are needed to bridge the gap between bytes and bedside.

Project description:BackgroundSystematic review is an indispensable tool for optimal evidence collection and evaluation in evidence-based medicine. However, the explosive increase of the original literatures makes it difficult to accomplish critical appraisal and regular update. Artificial intelligence (AI) algorithms have been applied to automate the literature screening procedure in medical systematic reviews. In these studies, different algorithms were used and results with great variance were reported. It is therefore imperative to systematically review and analyse the developed automatic methods for literature screening and their effectiveness reported in current studies.MethodsAn electronic search will be conducted using PubMed, Embase, ACM Digital Library, and IEEE Xplore Digital Library databases, as well as literatures found through supplementary search in Google scholar, on automatic methods for literature screening in systematic reviews. Two reviewers will independently conduct the primary screening of the articles and data extraction, in which nonconformities will be solved by discussion with a methodologist. Data will be extracted from eligible studies, including the basic characteristics of study, the information of training set and validation set, and the function and performance of AI algorithms, and summarised in a table. The risk of bias and applicability of the eligible studies will be assessed by the two reviewers independently based on Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). Quantitative analyses, if appropriate, will also be performed.DiscussionAutomating systematic review process is of great help in reducing workload in evidence-based practice. Results from this systematic review will provide essential summary of the current development of AI algorithms for automatic literature screening in medical evidence synthesis and help to inspire further studies in this field.Systematic review registrationPROSPERO CRD42020170815 (28 April 2020).

Project description:BackgroundThe mechanisms of the occurrence and progression of dilated cardiomyopathy are still unclear and further exploration is needed. The upgrading of programming languages and the improvement of biological databases have created conditions for us to explore the structural and functional information of biological molecules at the nucleic acid and protein levels, screen key pathogenic genes, and elucidate pathogenic mechanisms. This study aimed to screen key pathogenic genes using machine learning algorithms and explore the correlation between key genes and immune microenvironment through transcriptome sequencing data sets of myocardial samples from patients with dilated cardiomyopathy, providing new ideas for elucidating the pathogenesis of the disease.MethodsThe transcriptome sequencing data sets of heart tissue from patients with dilated cardiomyopathy were downloaded from the Gene Expression Omnibus (GEO) database (GSE29819 and GSE21610). Differentially expressed genes (DEGs) were screened between pathological and normal tissues. The key genes were screened using least absolute shrinkage and selection operator (LASSO) regression analysis and random forest tree algorithms. The diagnostic efficiency of the key genes for the disease was evaluated using the receiver operating characteristic (ROC) curve.ResultsCompared with the normal heart tissue (control group) samples, there were 213 DEGs in the heart tissue samples of patients with dilated cardiomyopathy (treat group), including 101 upregulated and 102 downregulated genes. CCL5 and CTGF were highly expressed in the treat group compared to the control group. The ROC curve showed that the areas under the curve (AUCs) of CCL5 and CTGF were 0.821 and 0.902, respectively (P<0.05). In the treat group samples, CCL5 was positively correlated with the infiltration content of most immune cell subtypes.ConclusionsCCL5 and CTGF are key disease-causing genes in dilated cardiomyopathy and have good diagnostic efficiency for the disease. CCL5 and CTGF may be related to immune cell enrichment and myocardial fibrosis, respectively.