Dataset Information

Suboptimal Quality and High Risk of Bias in Diagnostic Test Accuracy Studies at Chest Radiography and CT in the Acute Setting of the COVID-19 Pandemic: A Systematic Review.

ABSTRACT:

Purpose

To synthesize the literature on diagnostic test accuracy of chest radiography, CT, and US for the diagnosis of coronavirus disease 2019 (COVID-19) in patients suspected of having COVID-19 in a hospital setting and evaluate the extent of suboptimal reporting and risk of bias.

Materials and methods

A systematic search was performed (April 26, 2020) in EMBASE, PubMed, and Cochrane to identify chest radiographic, CT, or US studies in adult patients suspected of having COVID-19, using reverse-transcription polymerase chain reaction test or clinical consensus as the standard of reference. Two × two contingency tables were reconstructed, and test sensitivity, specificity, positive predictive values, and negative predictive values were recalculated. Reporting quality was evaluated by adherence to the Standards for Reporting of Diagnostic Accuracy Studies (STARD), and risk of bias was evaluated by adherence to the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2).

Results

Thirteen studies were eligible (CT = 12; chest radiography = 1; US = 0). Recalculated CT sensitivity and specificity ranged between 0.57 and 0.97, and 0.37 and 0.94, respectively, and positive predictive values and negative predictive values ranged between 0.59 and 0.92 and 0.57 and 0.96, respectively. On average, studies complied with only 35% of the STARD-guideline items. No study scored low risk of bias for all QUADAS-2 domains (patient selection, index test, reference test, and flow and timing). High risk of bias in more than one domain was scored in 10 of 13 studies (77%).

Conclusion

Reported CT test accuracy for COVID-19 diagnosis varies substantially. The validity and generalizability of these findings is complicated by poor adherence to reporting guidelines and high risk of bias, which are most likely due to the need for urgent publication of findings in the first months of the COVID-19 pandemic.Supplemental material is available for this article.© RSNA, 2020.

SUBMITTER: Sucha D

PROVIDER: S-EPMC7393956 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ObjectivesTo assess inter-reader agreements and diagnostic accuracy of chest CT to identify COVID-19 pneumonia in patients with intermediate clinical probability during an acute disease outbreak.MethodsFrom March 20 to April 8, 319 patients (mean age 62.3 years old) consecutive patients with an intermediate clinical probability of COVID-19 pneumonia underwent a chest CT scan. Two independent chest radiologists blinded to clinical information and RT-PCR results retrospectively reviewed and classified images on a 1-5 confidence level scale for COVID-19 pneumonia. Agreements between radiologists were assessed with kappa statistics. Diagnostic accuracy of chest CT compared with RT-PCR assay and patient outcomes was measured using receiver operating characteristics (ROC). Positive predictive value (PPV) and negative predictive value (NPV) for COVID-19 pneumonia were calculated.ResultsInter-observer agreement for highly probable (kappa: 0.83 [p < .001]) and highly probable or probable (kappa: 0.82 [p < .001]) diagnosis of COVID-19 pneumonia was very good. RT-PCR tests performed in 307 patients were positive in 174 and negative in 133. The areas under the curve (AUC) were 0.94 and 0.92 respectively. With a disease prevalence of 61.2%, PPV were 95.9% and 94.3%, and NPV 84.4% and 77.1%.ConclusionDuring acute COVID-19 outbreak, chest CT scan may be used for triage of patients with intermediate clinical probability with very good inter-observer agreements and diagnostic accuracy.Key points• Concordances between two chest radiologists to diagnose or exclude a COVID-19 pneumonia in 319 consecutive patients with intermediate clinical probability were very good (kappa: 0.82; p < .001). • When compared with RT-PCR results and patient outcomes, the diagnostic accuracy of CT to identify COVID-19 pneumonia was high for both radiologists (AUC: 0.94 and 0.92). • With a disease prevalence of 61.2% in the studied population, the positive predictive values of CT for diagnosing COVID-19 pneumonia were 95.9% and 94.3% with negative predictive values of 84.4% and 77.1%.

Project description:ObjectiveThe chest X-ray (CXR) is the most readily available and common imaging modality for the assessment of pneumonia. However, detecting pneumonia from chest radiography is a challenging task, even for experienced radiologists. An artificial intelligence (AI) model might help to diagnose pneumonia from CXR more quickly and accurately. We aim to develop an AI model for pneumonia from CXR images and to evaluate diagnostic performance with external dataset.MethodsTo train the pneumonia model, a total of 157,016 CXR images from the National Institutes of Health (NIH) and the Korean National Tuberculosis Association (KNTA) were used (normal vs. pneumonia = 120,722 vs.36,294). An ensemble model of two neural networks with DenseNet classifies each CXR image into pneumonia or not. To test the accuracy of the models, a separate external dataset of pneumonia CXR images (n = 212) from a tertiary university hospital (Gachon University Gil Medical Center GUGMC, Incheon, South Korea) was used; the diagnosis of pneumonia was based on both the chest CT findings and clinical information, and the performance evaluated using the area under the receiver operating characteristic curve (AUC). Moreover, we tested the change of the AI probability score for pneumonia using the follow-up CXR images (7 days after the diagnosis of pneumonia, n = 100).ResultsWhen the probability scores of the models that have a threshold of 0.5 for pneumonia, two models (models 1 and 4) having different pre-processing parameters on the histogram equalization distribution showed best AUC performances of 0.973 and 0.960, respectively. As expected, the ensemble model of these two models performed better than each of the classification models with 0.983 AUC. Furthermore, the AI probability score change for pneumonia showed a significant difference between improved cases and aggravated cases (Δ = -0.06 ± 0.14 vs. 0.06 ± 0.09, for 85 improved cases and 15 aggravated cases, respectively, P = 0.001) for CXR taken as a 7-day follow-up.ConclusionsThe ensemble model combined two different classification models for pneumonia that performed at 0.983 AUC for an external test dataset from a completely different data source. Furthermore, AI probability scores showed significant changes between cases of different clinical prognosis, which suggest the possibility of increased efficiency and performance of the CXR reading at the diagnosis and follow-up evaluation for pneumonia.