Training sample selection: Impact on screening automation in diagnostic test accuracy reviews.
Ontology highlight
ABSTRACT:
SUBMITTER: van Altena AJ
PROVIDER: S-EPMC9292892 | biostudies-literature |
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:BackgroundThe large and increasing number of new studies published each year is making literature identification in systematic reviews ever more time-consuming and costly. Technological assistance has been suggested as an alternative to the conventional, manual study identification to mitigate the cost, but previous literature has mainly evaluated methods in terms of recall (search sensitivity) and workload reduction. There is a need to also evaluate whether screening prioritization methods leads to the same results and conclusions as exhaustive manual screening. In this study, we examined the impact of one screening prioritization method based on active learning on sensitivity and specificity estimates in systematic reviews of diagnostic test accuracy.MethodsWe simulated the screening process in 48 Cochrane reviews of diagnostic test accuracy and re-run 400 meta-analyses based on a least 3 studies. We compared screening prioritization (with technological assistance) and screening in randomized order (standard practice without technology assistance). We examined if the screening could have been stopped before identifying all relevant studies while still producing reliable summary estimates. For all meta-analyses, we also examined the relationship between the number of relevant studies and the reliability of the final estimates.ResultsThe main meta-analysis in each systematic review could have been performed after screening an average of 30% of the candidate articles (range 0.07 to 100%). No systematic review would have required screening more than 2308 studies, whereas manual screening would have required screening up to 43,363 studies. Despite an average 70% recall, the estimation error would have been 1.3% on average, compared to an average 2% estimation error expected when replicating summary estimate calculations.ConclusionScreening prioritization coupled with stopping criteria in diagnostic test accuracy reviews can reliably detect when the screening process has identified a sufficient number of studies to perform the main meta-analysis with an accuracy within pre-specified tolerance limits. However, many of the systematic reviews did not identify a sufficient number of studies that the meta-analyses were accurate within a 2% limit even with exhaustive manual screening, i.e., using current practice.
| S-EPMC6819363 | biostudies-literature
Project description:In 1996, shortly after the founding of The Cochrane Collaboration, leading figures in test evaluation research established a Methods Group to focus on the relatively new and rapidly evolving methods for the systematic review of studies of diagnostic tests. Seven years later, the Collaboration decided it was time to develop a publication format and methodology for Diagnostic Test Accuracy (DTA) reviews, as well as the software needed to implement these reviews in The Cochrane Library. A meeting hosted by the German Cochrane Centre in 2004 brought together key methodologists in the area, many of whom became closely involved in the subsequent development of the methodological framework for DTA reviews. DTA reviews first appeared in The Cochrane Library in 2008 and are now an integral part of the work of the Collaboration.
| S-EPMC3851548 | biostudies-literature
Project description:BackgroundSystematic review is an indispensable tool for optimal evidence collection and evaluation in evidence-based medicine. However, the explosive increase of the original literatures makes it difficult to accomplish critical appraisal and regular update. Artificial intelligence (AI) algorithms have been applied to automate the literature screening procedure in medical systematic reviews. In these studies, different algorithms were used and results with great variance were reported. It is therefore imperative to systematically review and analyse the developed automatic methods for literature screening and their effectiveness reported in current studies.MethodsAn electronic search will be conducted using PubMed, Embase, ACM Digital Library, and IEEE Xplore Digital Library databases, as well as literatures found through supplementary search in Google scholar, on automatic methods for literature screening in systematic reviews. Two reviewers will independently conduct the primary screening of the articles and data extraction, in which nonconformities will be solved by discussion with a methodologist. Data will be extracted from eligible studies, including the basic characteristics of study, the information of training set and validation set, and the function and performance of AI algorithms, and summarised in a table. The risk of bias and applicability of the eligible studies will be assessed by the two reviewers independently based on Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). Quantitative analyses, if appropriate, will also be performed.DiscussionAutomating systematic review process is of great help in reducing workload in evidence-based practice. Results from this systematic review will provide essential summary of the current development of AI algorithms for automatic literature screening in medical evidence synthesis and help to inspire further studies in this field.Systematic review registrationPROSPERO CRD42020170815 (28 April 2020).
| S-EPMC8760775 | biostudies-literature
Project description:ObjectivesTo compare the accuracy of trained level 1 diabetic retinopathy (DR) graders (nurses, endocrinologists and one general practitioner), level 2 graders (midlevel ophthalmologists) and level 3 graders (senior ophthalmologists) in Vietnam against a reference standard from the UK and assess the impact of supplementary targeted grader training.DesignDiagnostic test accuracy study.SettingSecondary care hospitals in Southern Vietnam.ParticipantsDR training was delivered to Vietnamese graders in February 2018 by National Health Service (NHS) UK graders. Two-field retinal images (412 patient images) were graded by 14 trained graders in Vietnam between August and October 2018 and then regraded retrospectively by an NHS-certified reference standard UK optometrist (phase I). Further DR training based on phase I results was delivered to graders in November 2019. After training, a randomised subset of images from January to October 2020 (115 patient images) was graded by six of the original cohort (phase II). The reference grader regraded all images from phase I and II retrospectively in masked fashion.Primary and secondary outcome measuresSensitivity was calculated at the two different time points, and χ2 was used to test significance.ResultsIn phase I, the sensitivity for detecting any DR for all grader groups in Vietnam was low (41.8-42.2%) and improved in phase II after additional training was delivered (51.3-87.2%). The greatest improvement was seen among level 1 graders (p<0.001), and the lowest improvement was observed among level 3 graders (p=0.326). There was a statistically significant improvement in sensitivity for detecting referable DR and referable diabetic macular oedema between all grader levels. The post-training values ranged from 40.0 to 61.5% (including ungradable images) and 55.6%-90.0% (excluding ungradable images).ConclusionsThis study demonstrates that targeted training interventions can improve accuracy of DR grading. These findings have important implications for improving service delivery in DR screening programmes in low-resource settings.
| S-EPMC9472142 | biostudies-literature
Project description:The Spurling test, although a highly specific provocative test of the cervical spine in cervical radiculopathy (CR), has low to moderate sensitivity. Thus, we introduced the neck tornado test (NTT) to examine the neck and the cervical spine in CR.The aim of this study was to introduce a new provocative test, the NTT, and compare the diagnostic accuracy with a widely accepted provocative test, the Spurling test.Retrospective study.Medical records of 135 subjects with neck pain (CR, n = 67; without CR, n = 68) who had undergone cervical spine magnetic resonance imaging and been referred to the pain clinic between September 2014 and August 2015 were reviewed. Both the Spurling test and NTT were performed in all patients by expert examiners. Sensitivity, specificity, and accuracy were compared for both the Spurling test and the NTT.The sensitivity of the Spurling test and the NTT was 55.22% and 85.07% (P < 0.0001); specificity, 98.53% and 86.76% (P = 0.0026); accuracy, 77.04% and 85.93% (P = 0.0423), respectively.The NTT is more sensitive with superior diagnostic accuracy for CR diagnosed by magnetic resonance imaging than the Spurling test.
| S-EPMC5562117 | biostudies-other
Project description:The standard item response theory (IRT) model assumption of a single homogenous population may be violated in real data. Mixture extensions of IRT models have been proposed to account for latent heterogeneous populations, but these models are not designed to handle multilevel data structures. Ignoring the multilevel structure is problematic as it results in lower-level units aggregated with higher-level units and yields less accurate results, because of dependencies in the data. Multilevel data structures cause such dependencies between levels but can be modeled in a straightforward way in multilevel mixture IRT models. An important step in the use of multilevel mixture IRT models is the fit of the model to the data. This fit is often determined based on relative fit indices. Previous research on mixture IRT models has shown that performances of these indices and classification accuracy of these models can be affected by several factors including percentage of class-variant items, number of items, magnitude and size of clusters, and mixing proportions of latent classes. As yet, no studies appear to have been reported examining these issues for multilevel extensions of mixture IRT models. The current study aims to investigate the effects of several features of the data on the accuracy of model selection and parameter recovery. Results are reported on a simulation study designed to examine the following features of the data: percentages of class-variant items (30, 60, and 90%), numbers of latent classes in the data (with from 1 to 3 latent classes at level 1 and 1 and 2 latent classes at level 2), numbers of items (10, 30, and 50), numbers of clusters (50 and 100), cluster size (10 and 50), and mixing proportions [equal (0.5 and 0.5) vs. non-equal (0.25 and 0.75)]. Simulation results indicated that multilevel mixture IRT models resulted in less accurate estimates when the number of clusters and the cluster size were small. In addition, mean Root mean square error (RMSE) values increased as the percentage of class-variant items increased and parameters were recovered more accurately under the 30% class-variant item conditions. Mixing proportion type (i.e., equal vs. unequal latent class sizes) and numbers of items (10, 30, and 50), however, did not show any clear pattern. Sample size dependent fit indices BIC, CAIC, and SABIC performed poorly for the smaller level-1 sample size. For the remaining conditions, the SABIC index performed better than other fit indices.
| S-EPMC7033749 | biostudies-literature
Project description:Objective: The diagnostic efficiency of the quantitative fecal immunochemical test (qFIT) has large variations in colorectal cancer (CRC) screening. We aimed to explore whether the practical sample collection operant training could improve the diagnostic accuracy of the qFIT in CRC screening. Methods: Moderate-/high-risk individuals aged 50-75 years old were invited to participate in a prospective observational study between July 2020 and March 2021. Participants took a qFIT sample without fecal sample collection operant training in advance and then completed another qFIT sample after the operant training. The primary outcome was the sensitivity and specificity of the qFITs for CRC and advanced colorectal neoplasia (ACRN). The secondary outcome was the difference in the area under the curves (AUCs) and the concentrations of the fecal hemoglobin (Hb) between the qFIT without and after the operant training. Results: Out of 913 patients, 81 (8.9%) patients had ACRN, including 25 (2.7%) patients with CRC. For CRC, the sensitivities of the qFIT without and after the operant training at 10 μg/g were 80.4 and 100.0%, respectively, and the specificities were 90.1 and 88.4%, respectively. For ACRN, the sensitivities were 49.4 and 69.1% and the specificities were 91.7 and 91.3%, respectively. The AUC of the qFIT after the operant training was significantly higher than that without the operant training for CRC (p = 0.027) and ACRN (p = 0.001). After the operant training, the concentration of the fecal Hb was significantly higher than that without the operant training (p = 0.009) for ACRN, but there was no significant difference for CRC (p = 0.367). Conclusion: Practical sample collection operant training improves the diagnostic accuracy of the qFIT, which increases the detection of the low concentrations of fecal Hb. Improving the quality of the sample collection could contribute to the diagnostic efficiency of the qFIT in CRC screening.
| S-EPMC8575757 | biostudies-literature
Project description:BACKGROUND:A pretest probability must be selected to calculate data to help clinicians, guideline boards and policy makers interpret diagnostic accuracy parameters. When multiple analyses for the same target condition are compared, identical pretest probabilities might be selected to facilitate the comparison. Some pretest probabilities may lead to exaggerations of the patient harms or benefits, and guidance on how and why to select a specific pretest probability is minimally described. Therefore, the aim of this study was to assess the data sources and methods used in Cochrane diagnostic test accuracy (DTA) reviews for determining pretest probabilities to facilitate the interpretation of DTA parameters. A secondary aim was to assess the use of identical pretest probabilities to compare multiple meta-analyses within the same target condition. METHODS:Cochrane DTA reviews presenting at least one meta-analytic estimate of the sensitivity and/or specificity as a primary analysis published between 2008 and January 2018 were included. Study selection and data extraction were performed by one author and checked by other authors. Observed data sources (e.g. studies in the review, or external sources) and methods to select pretest probabilities (e.g. median) were categorized. RESULTS:Fifty-nine DTA reviews were included, comprising of 308 meta-analyses. A pretest probability was used in 148 analyses. Authors used included studies in the DTA review, external sources, and author consensus as data sources for the pretest probability. Measures of central tendency with or without a measure of dispersion were used to determine the pretest probabilities, with the median most commonly used. Thirty-two target conditions had at least one identical pretest probability for all of the meta-analyses within their target condition. About half of the used identical pretest probabilities were inside the prevalence ranges from all analyses within a target condition. CONCLUSIONS:Multiple sources and methods were used to determine (identical) pretest probabilities in Cochrane DTA reviews. Indirectness and severity of downstream consequences may influence the acceptability of the certainty in calculated data with pretest probabilities. Consider: whether to present normalized frequencies, the influence of pretest probabilities on normalized frequencies, and whether to use identical pretest probabilities for meta-analyses in a target condition.
| S-EPMC7161259 | biostudies-literature
Project description:The Cochrane Collaboration says that the Cochrane handbook for diagnostic test accuracy reviews (DTAR) is currently in development as per the Cochrane Collaboration. This implies that the methodology of systematic reviews (SR) of diagnostic test accuracy is still a matter of debate. At this point, comparison of methodologies for SR in case of interventions as against diagnostics would be helpful to understand DTAR.
| S-EPMC4183058 | biostudies-literature
Project description:BACKGROUND:Robust diagnosis of dementia requires an understanding of the accuracy of the available diagnostic tests. Informant questionnaires are frequently used to assess for dementia in clinical practice. Recent systematic reviews have sought to establish the diagnostic test accuracy of various dementia informant screening tools. However, most reviews to date have focused on a single diagnostic tool and this does not address which tool is 'best'. A key aim of the overview of systematic reviews is to present a disparate evidence base in a single, easy to access platform. METHODS:We will conduct an overview of systematic reviews in which we 'review the systematic reviews' of diagnostic test accuracy studies evaluating informant questionnaires for dementia. As an overview of systematic reviews of test accuracy is a relatively novel approach, we will use this review to explore methods for visual representation of complex data, for highlighting evidence gaps and for indirect comparative analyses. We will create a list of informant tools by consulting with dementia experts. We will search 6 databases (EMBASE (OVID); Health and Psychosocial Instruments (OVID); Medline (OVID); CINAHL (EBSCO); PSYCHinfo (EBSCO) and the PROSPERO registry of review protocols) to identify systematic reviews that describe the diagnostic test accuracy of informant questionnaires for dementia. We will assess review quality using the AMSTAR-2 (Assessment of Multiple Systematic Reviews) and assess reporting quality using PRISMA-DTA (Preferred Reporting Items for Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies) checklists. We will collate the identified reviews to create an 'evidence map' that highlights where evidence does and does not exist in relation to informant questionnaires. We will pool sensitivity and specificity data via meta-analysis to generate a diagnostic test accuracy summary statistic for each informant questionnaire. If data allow, we will perform a statistical comparison of the diagnostic test accuracy of each informant questionnaire using a network approach. DISCUSSION:Our overview of systematic reviews will provide a concise summary of the diagnostic test accuracy of informant tools and highlight areas where evidence is currently lacking in this regard. It will also apply network meta-analysis techniques to a new area.
| S-EPMC7694897 | biostudies-literature