Dataset Information

Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy.

ABSTRACT:

Background

The large and increasing number of new studies published each year is making literature identification in systematic reviews ever more time-consuming and costly. Technological assistance has been suggested as an alternative to the conventional, manual study identification to mitigate the cost, but previous literature has mainly evaluated methods in terms of recall (search sensitivity) and workload reduction. There is a need to also evaluate whether screening prioritization methods leads to the same results and conclusions as exhaustive manual screening. In this study, we examined the impact of one screening prioritization method based on active learning on sensitivity and specificity estimates in systematic reviews of diagnostic test accuracy.

Methods

We simulated the screening process in 48 Cochrane reviews of diagnostic test accuracy and re-run 400 meta-analyses based on a least 3 studies. We compared screening prioritization (with technological assistance) and screening in randomized order (standard practice without technology assistance). We examined if the screening could have been stopped before identifying all relevant studies while still producing reliable summary estimates. For all meta-analyses, we also examined the relationship between the number of relevant studies and the reliability of the final estimates.

Results

The main meta-analysis in each systematic review could have been performed after screening an average of 30% of the candidate articles (range 0.07 to 100%). No systematic review would have required screening more than 2308 studies, whereas manual screening would have required screening up to 43,363 studies. Despite an average 70% recall, the estimation error would have been 1.3% on average, compared to an average 2% estimation error expected when replicating summary estimate calculations.

Conclusion

Screening prioritization coupled with stopping criteria in diagnostic test accuracy reviews can reliably detect when the screening process has identified a sufficient number of studies to perform the main meta-analysis with an accuracy within pre-specified tolerance limits. However, many of the systematic reviews did not identify a sufficient number of studies that the meta-analyses were accurate within a 2% limit even with exhaustive manual screening, i.e., using current practice.

SUBMITTER: Norman CR

PROVIDER: S-EPMC6819363 | biostudies-literature | 2019 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy.

Norman Christopher R CR Leeflang Mariska M G MMG Porcher Raphaël R Névéol Aurélie A

Systematic reviews 20191028 1

<h4>Background</h4>The large and increasing number of new studies published each year is making literature identification in systematic reviews ever more time-consuming and costly. Technological assistance has been suggested as an alternative to the conventional, manual study identification to mitigate the cost, but previous literature has mainly evaluated methods in terms of recall (search sensitivity) and workload reduction. There is a need to also evaluate whether screening prioritization met ...[more]

PMID: 31661028

Similar Datasets

Project description:BackgroundSystematic review is an indispensable tool for optimal evidence collection and evaluation in evidence-based medicine. However, the explosive increase of the original literatures makes it difficult to accomplish critical appraisal and regular update. Artificial intelligence (AI) algorithms have been applied to automate the literature screening procedure in medical systematic reviews. In these studies, different algorithms were used and results with great variance were reported. It is therefore imperative to systematically review and analyse the developed automatic methods for literature screening and their effectiveness reported in current studies.MethodsAn electronic search will be conducted using PubMed, Embase, ACM Digital Library, and IEEE Xplore Digital Library databases, as well as literatures found through supplementary search in Google scholar, on automatic methods for literature screening in systematic reviews. Two reviewers will independently conduct the primary screening of the articles and data extraction, in which nonconformities will be solved by discussion with a methodologist. Data will be extracted from eligible studies, including the basic characteristics of study, the information of training set and validation set, and the function and performance of AI algorithms, and summarised in a table. The risk of bias and applicability of the eligible studies will be assessed by the two reviewers independently based on Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). Quantitative analyses, if appropriate, will also be performed.DiscussionAutomating systematic review process is of great help in reducing workload in evidence-based practice. Results from this systematic review will provide essential summary of the current development of AI algorithms for automatic literature screening in medical evidence synthesis and help to inspire further studies in this field.Systematic review registrationPROSPERO CRD42020170815 (28 April 2020).

Project description:BackgroundIndividual patient data meta-analyses (IPD-MA) are regarded as the gold standard for systematic reviews, which also applies to systematic reviews of diagnostic test accuracy (DTA) studies. An increasing number of DTA systematic reviews with IPD-MA have been published in recent years, but there is much variation in how these IPD-MA were performed. A number of existing methods were found, but there is no consensus as to which methods are preferred as the standard methods for statistical analysis in DTA IPD-MA.ObjectivesTo create a web-based tool which integrates recommended statistical analyses for DTA IPD-MA, and allows researchers to analyse the data and visualize the results with interactive plots.MethodsA systematic methodological review was performed to identify statistical analyses and data visualization methods used in DTA IPD-MA. Methods were evaluated by the authors and recommended analyses were integrated into the IPDmada tool which is freely available online with the user interface developed with R Shiny package.ResultsIPDmada allows users to upload their own data, perform the meta-analysis with both continuous and dichotomized tests, and incorporate individual level covariate-adjusted analysis. All tables and figures can be exported as .csv or .pdf files. A hypothetical dataset was used to illustrate the application of IPDmada.ConclusionsIPDmada will be very helpful to researchers doing DTA IPD-MA, since it not only facilitates the statistical analysis but also provides a standard framework. The introduction of IPDmada will harmonize the methods used in DTA IPD-MA and ensure the quality of such analyses.HighlightsIPDmada is a newly developed web-based tool for performing statistical analysis of individual patient data meta-analysis of diagnostic accuracy and visualizing the results. The tool is freely available to all the researchers, and requiring no installation of statistical software/packages. The tool has an user-friendly interface, and allows meta-analysis on both dichotomized and continuous test results. Researchers can easily use this tool to investigate the threshold effect and covariate effect on the summary accuracy. The introduction and implementation of IPDmada will serve as a useful tool for DTA IPD-MA and increase the quality of such studies.

Project description:ImportanceRed reflex testing is a simple and inexpensive method implemented in many countries as an important part of infant screening for ocular pathologies.ObjectivesTo review the literature on the diagnostic accuracy of the red reflex test in infant screening for ocular pathologies and to perform meta-analyses to provide summary estimates.Data sourcesThe following literature databases were searched for English-language, peer-reviewed literature, published until April 19, 2020: Cochrane Central, PubMed/MEDLINE, Embase, Web of Science Core Collection, BIOSIS Previews, Current Contents Connect, Data Citation Index, Derwent Innovations Index, KCI-Korean Journal Database, Russian Science Citation Index, SciELO Citation Index, CINAHL (Cumulative Index to Nursing and Allied Health Literature), and ClinicalTrials.gov.Study selectionEligibility criteria were defined according to population (studies of consecutively screened infants), exposure (red reflex or Brückner test as the index test), comparator (any ophthalmological examination), and study type (any study with diagnostic test accuracy data).Data extraction and synthesisThe Preferred Reporting Items for Systematic Reviews and Meta-Analysis of Diagnostic Test Accuracy Studies (PRISMA-DTA) guidelines were followed. Data were extracted independently by 2 authors. For summary estimates of diagnostic test accuracy, the hierarchical summary receiver operating characteristics curve was used. Prevalence of ocular pathologies was introduced for a prevalence meta-analysis, which was then used in calculations of diagnostic accuracy of the red reflex test when applied in infant screening.Main outcomes and measuresTrue-positive, false-positive, true-negative, and false-negative findings; sensitivity; specificity; and positive and negative predictive values.ResultsIn this meta-analysis, 8713 unique infants from 5 unique studies were eligible for qualitative and quantitative review. All studies used the red reflex test without pupillary dilation and were compared with a reference test performed with pupillary dilation. For any ocular pathology, an estimated sensitivity of 7.5% (95% CI, 7.4%-7.5%) and specificity of 97.5% (95% CI, 97.5%-97.5%) was found. Focusing on ocular pathologies that required a medical or surgical intervention, sensitivity improved to 17.5% (95% CI, 0.8%-84.8%) and specificity remained high at 97.6% (95% CI, 87.7%-99.6%).Conclusions and relevanceThese findings suggest that an abnormal red reflex finding most likely reflects an underlying ocular pathology. However, a normal red reflex finding during screening does not exclude ocular disease.

Dataset Information

Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy.

Background

Methods

Results

Conclusion

Publications

Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets