Dataset Information

Abstract screening using the automated tool Rayyan: results of effectiveness in three diagnostic test accuracy systematic reviews.

ABSTRACT:

Objective

To evaluate the performance of the automated abstract screening tool Rayyan.

Methods

The records obtained from the search for three systematic reviews were manually screened in four stages. At the end of each stage, Rayyan was used to predict the eligibility score for the remaining records. At two different thresholds (≤2.5 and < 2.5 for exclusion of a record) Rayyan-generated ratings were compared with the decisions made by human reviewers in the manual screening process and the tool's accuracy metrics were calculated.

Results

Two thousand fifty-four records were screened manually, of which 379 were judged to be eligible for full-text assessment, and 112 were eventually included in the final review. For finding records eligible for full-text assessment, at the threshold of < 2.5 for exclusion, Rayyan managed to achieve sensitivity values of 97-99% with specificity values of 19-58%, while at the threshold of ≤2.5 for exclusion it had a specificity of 100% with sensitivity values of 1-29%. For the task of finding eligible reports for inclusion in the final review, almost similar results were obtained.

Discussion

At the threshold of < 2.5 for exclusion, Rayyan managed to be a reliable tool for excluding ineligible records, but it was not much reliable for finding eligible records. We emphasize that this study was conducted on diagnostic test accuracy reviews, which are more difficult to screen due to inconsistent terminology.

SUBMITTER: Valizadeh A

PROVIDER: S-EPMC9161508 | biostudies-literature | 2022 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Abstract screening using the automated tool Rayyan: results of effectiveness in three diagnostic test accuracy systematic reviews.

Valizadeh Amir A Moassefi Mana M Nakhostin-Ansari Amin A Hosseini Asl Seyed Hossein SH Saghab Torbati Mehrnush M Aghajani Reyhaneh R Maleki Ghorbani Zahra Z Faghani Shahriar S

BMC medical research methodology 20220602 1

<h4>Objective</h4>To evaluate the performance of the automated abstract screening tool Rayyan.<h4>Methods</h4>The records obtained from the search for three systematic reviews were manually screened in four stages. At the end of each stage, Rayyan was used to predict the eligibility score for the remaining records. At two different thresholds (≤2.5 and < 2.5 for exclusion of a record) Rayyan-generated ratings were compared with the decisions made by human reviewers in the manual screening proces ...[more]

PMID: 35655155

Similar Datasets

Project description:BackgroundWe evaluated the benefits and risks of using the Abstrackr machine learning (ML) tool to semi-automate title-abstract screening and explored whether Abstrackr's predictions varied by review or study-level characteristics.MethodsFor a convenience sample of 16 reviews for which adequate data were available to address our objectives (11 systematic reviews and 5 rapid reviews), we screened a 200-record training set in Abstrackr and downloaded the relevance (relevant or irrelevant) of the remaining records, as predicted by the tool. We retrospectively simulated the liberal-accelerated screening approach. We estimated the time savings and proportion missed compared with dual independent screening. For reviews with pairwise meta-analyses, we evaluated changes to the pooled effects after removing the missed studies. We explored whether the tool's predictions varied by review and study-level characteristics.ResultsUsing the ML-assisted liberal-accelerated approach, we wrongly excluded 0 to 3 (0 to 14%) records that were included in the final reports, but saved a median (IQR) 26 (9, 42) h of screening time. One missed study was included in eight pairwise meta-analyses in one systematic review. The pooled effect for just one of those meta-analyses changed considerably (from MD (95% CI) - 1.53 (- 2.92, - 0.15) to - 1.17 (- 2.70, 0.36)). Of 802 records in the final reports, 87% were correctly predicted as relevant. The correctness of the predictions did not differ by review (systematic or rapid, P = 0.37) or intervention type (simple or complex, P = 0.47). The predictions were more often correct in reviews with multiple (89%) vs. single (83%) research questions (P = 0.01), or that included only trials (95%) vs. multiple designs (86%) (P = 0.003). At the study level, trials (91%), mixed methods (100%), and qualitative (93%) studies were more often correctly predicted as relevant compared with observational studies (79%) or reviews (83%) (P = 0.0006). Studies at high or unclear (88%) vs. low risk of bias (80%) (P = 0.039), and those published more recently (mean (SD) 2008 (7) vs. 2006 (10), P = 0.02) were more often correctly predicted as relevant.ConclusionOur screening approach saved time and may be suitable in conditions where the limited risk of missing relevant records is acceptable. Several of our findings are paradoxical and require further study to fully understand the tasks to which ML-assisted screening is best suited. The findings should be interpreted in light of the fact that the protocol was prepared for the funder, but not published a priori. Because we used a convenience sample, the findings may be prone to selection bias. The results may not be generalizable to other samples of reviews, ML tools, or screening approaches. The small number of missed studies across reviews with pairwise meta-analyses hindered strong conclusions about the effect of missed studies on the results and conclusions of systematic reviews.

Project description:Introduction Systemic lupus erythematosus (SLE) is a chronic autoimmune disease with multiorgan inflammatory involvement and a mortality rate that is 2.6-fold higher than individuals of the same age and sex in the general population. Approximately 50% of patients with SLE develop renal impairment (lupus nephritis). Delayed diagnosis of lupus nephritis is associated with a higher risk of progression to end-stage renal disease, the need for replacement therapy, and mortality. The initial clinical manifestations of lupus nephritis are often discrete or absent and are usually detected through complementary tests. Although widely used in clinical practice, their accuracy is limited. A great scientific effort has been exerted towards searching for new, more sensitive, and specific biomarkers in recent years. Some systematic reviews have individually evaluated new serum and urinary biomarkers tested in patients with lupus nephritis. This overview aimed to summarize systematic reviews on the accuracy of novel serum and urinary biomarkers for diagnosing lupus nephritis in patients with SLE, discussing how our results can guide the clinical management of the disease and the direction of research in this area. Methods The research question is “What is the accuracy of the new serum and urinary biomarkers studied for the diagnosis of LN in patients with SLE?”. We searched for systematic reviews of observational studies evaluating the diagnostic accuracy of new serum or urinary biomarkers of lupus nephritis. The following databases were included: PubMed, EMBASE, BIREME/LILACS, Scopus, Web of Science, and Cochrane, including gray literature found via Google Scholar and PROQUEST. Two authors assessed the reviews for inclusion, data extraction, and assessment of the risk of bias (ROBIS tool). Results Ten SRs on the diagnostic accuracy of new serum and urinary BMs in LN were selected. The SRs evaluated 7 distinct BMs: (a) antibodies (anti-Sm, anti-RNP, and anti-C1q), (b) cytokines (TWEAK and MCP-1), (c) a chemokine (IP-10), and (d) an acute phase glycoprotein (NGAL), in a total of 20 review arms (9 that analyzed serum BMs, and 12 that analyzed BMs in urine). The population evaluated in the primary studies was predominantly adults. Two SRs included strictly adults, 5 reviews also included studies in the paediatric population, and 4 did not report the age groups. The results of the evaluation with the ROBIS tool showed that most of the reviews had a low overall risk of bias. Conclusions There are 10 SRs of evidence relating to the diagnostic accuracy of serum and urinary biomarkers for lupus nephritis. Among the BMs evaluated, anti-C1q, urinary MCP-1, TWEAK, and NGAL stood out, highlighting the need for additional research, especially on LN diagnostic panels, and attempting to address methodological issues within diagnostic accuracy research. This would allow for a better understanding of their usefulness and possibly validate their clinical use in the future. Registration This project is registered on the International Prospective Registry of Systematic Reviews (PROSPERO) database (CRD42020196693).

Project description:ImportanceSystematic reviews of medical imaging diagnostic test accuracy (DTA) studies are affected by between-study heterogeneity due to a range of factors. Failure to appropriately assess the extent and causes of heterogeneity compromises the interpretability of systematic review findings.ObjectiveTo assess how heterogeneity has been examined in medical imaging DTA studies.Evidence reviewThe PubMed database was searched for systematic reviews of medical imaging DTA studies that performed a meta-analysis. The search was limited to the 40 journals with highest impact factor in the radiology, nuclear medicine, and medical imaging category in the InCites Journal Citation Reports of 2021 to reach a sample size of 200 to 300 included studies. Descriptive analysis was performed to characterize the imaging modality, target condition, type of meta-analysis model used, strategies for evaluating heterogeneity, and sources of heterogeneity identified. Multivariable logistic regression was performed to assess whether any factors were associated with at least 1 source of heterogeneity being identified in the included meta-analyses. Methodological quality evaluation was not performed. Data analysis occurred from October to December 2022.FindingsA total of 242 meta-analyses involving a median (range) of 987 (119-441 510) patients across a diverse range of disease categories and imaging modalities were included. The extent of heterogeneity was adequately described (ie, whether it was absent, low, moderate, or high) in 220 studies (91%) and was most commonly assessed using the I2 statistic (185 studies [76%]) and forest plots (181 studies [75%]). Heterogeneity was rated as moderate to high in 191 studies (79%). Of all included meta-analyses, 122 (50%) performed subgroup analysis and 87 (36%) performed meta-regression. Of the 242 studies assessed, 189 (78%) included 10 or more primary studies. Of these 189 studies, 60 (32%) did not perform meta-regression or subgroup analysis. Reasons for being unable to investigate sources of heterogeneity included inadequate reporting of primary study characteristics and a low number of included primary studies. Use of meta-regression was associated with identification of at least 1 source of variability (odds ratio, 1.90; 95% CI, 1.11-3.23; P = .02).Conclusions and relevanceIn this systematic review of assessment of heterogeneity in medical imaging DTA meta-analyses, most meta-analyses were impacted by a moderate to high level of heterogeneity, presenting interpretive challenges. These findings suggest that, despite the development and availability of more rigorous statistical models, heterogeneity appeared to be incomplete, inconsistently evaluated, or methodologically questionable in many cases, which lessened the interpretability of the analyses performed; comprehensive heterogeneity assessment should be addressed at the author level by improving personal familiarity with appropriate statistical methodology for assessing heterogeneity and involving biostatisticians and epidemiologists in study design, as well as at the editorial level, by mandating adherence to methodologic standards in primary DTA studies and DTA meta-analyses.

Dataset Information

Abstract screening using the automated tool Rayyan: results of effectiveness in three diagnostic test accuracy systematic reviews.

Objective

Methods

Results

Discussion

Publications

Abstract screening using the automated tool Rayyan: results of effectiveness in three diagnostic test accuracy systematic reviews.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets