Dataset Information

Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence.

ABSTRACT: BACKGROUND:The exponential growth of the biomedical literature necessitates investigating strategies to reduce systematic reviewer burden while maintaining the high standards of systematic review validity and comprehensiveness. METHODS:We compared the traditional systematic review screening process with (1) a review-of-reviews (ROR) screening approach and (2) a semi-automation screening approach using two publicly available tools (RobotAnalyst and AbstrackR) and different types of training sets (randomly selected citations subjected to dual-review at the title-abstract stage, highly curated citations dually reviewed at the full-text stage, and a combination of the two). We evaluated performance measures of sensitivity, specificity, missed citations, and workload burden RESULTS: The ROR approach for treatments of early-stage prostate cancer had a poor sensitivity (0.54) and studies missed by the ROR approach tended to be of head-to-head comparisons of active treatments, observational studies, and outcomes of physical harms and quality of life. Title and abstract screening incorporating semi-automation only resulted in a sensitivity of 100% at high levels of reviewer burden (review of 99% of citations). A highly curated, smaller-sized, training set (n = 125) performed similarly to a larger training set of random citations (n = 938). CONCLUSION:Two approaches to rapidly update SRs-review-of-reviews and semi-automation-failed to demonstrate reduced workload burden while maintaining an acceptable level of sensitivity. We suggest careful evaluation of the ROR approach through comparison of inclusion criteria and targeted searches to fill evidence gaps as well as further research of semi-automation use, including more study of highly curated training sets.

SUBMITTER: Reddy SM

PROVIDER: S-EPMC7574591 | biostudies-literature | 2020 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence.

Reddy Shivani M SM Patel Sheila S Weyrich Meghan M Fenton Joshua J Viswanathan Meera M

Systematic reviews 20201019 1

<h4>Background</h4>The exponential growth of the biomedical literature necessitates investigating strategies to reduce systematic reviewer burden while maintaining the high standards of systematic review validity and comprehensiveness.<h4>Methods</h4>We compared the traditional systematic review screening process with (1) a review-of-reviews (ROR) screening approach and (2) a semi-automation screening approach using two publicly available tools (RobotAnalyst and AbstrackR) and different types of ...[more]

PMID: 33076975

Similar Datasets

Project description:Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search MEDLINE, Institute of Electrical and Electronics Engineers (IEEE), arXiv, and the dblp computer science bibliography databases. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This iteration of the living review includes publications up to a cut-off date of 22 April 2020. Results: In total, 53 publications are included in this version of our review. Of these, 41 (77%) of the publications addressed extraction of data from abstracts, while 14 (26%) used full texts. A total of 48 (90%) publications developed and evaluated classifiers that used randomised controlled trials as the main target texts. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. A description of their datasets was provided by 49 publications (94%), but only seven (13%) made the data publicly available. Code was made available by 10 (19%) publications, and five (9%) implemented publicly available tools. Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of systematic review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. The lack of publicly available gold-standard data for evaluation, and lack of application thereof, makes it difficult to draw conclusions on which is the best-performing system for each data extraction target. With this living review we aim to review the literature continually.

Project description:BackgroundThe demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice.MethodsIn November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies.ResultsFrom 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs.ConclusionsAutomation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.

Project description:BackgroundA number of methods for deciding when a systematic review should be updated have been proposed, yet little is known about whether systematic reviews are updated more quickly when new evidence becomes available. Our aim was to examine the timing of systematic review updates relative to the availability of new evidence.MethodsWe performed a retrospective analysis of the update timing of systematic reviews published in the Cochrane Database of Systematic Reviews in 2010 relative to the availability of new trial evidence. We compared the update timing of systematic reviews with and without signals defined by the completion or publication of studies that were included in the updates.ResultsWe found 43% (293/682) systematic reviews were updated before June 2017, of which 204 included an updated primary outcome meta-analysis (median update time 35.4 months; IQR 25.5-54.0), 38% (77/204) added new trials, and 4% (8/204) reported a change in conclusion. In the 171 systematic reviews with reconcilable trial reporting information, we did not find a clear difference in update timing (p = 0.05) between the 15 systematic reviews with a publication signal (median 25.3 months; IQR 15.3-43.5) and the 156 systematic reviews without a publication signal (median 34.4 months; IQR 25.1-52.2). In the 145 systematic reviews with reconcilable trial completion information, we did not find a difference in update timing (p = 0.33) between the 15 systematic reviews with a trial completion signal (median 26.0 months; IQR 19.3-49.5) and the 130 systematic reviews without a trial completion signal (median 32.4 months; IQR 24.1 to 46.0).ConclusionA minority of 2010 Cochrane reviews were updated before June 2017 to incorporate evidence from new primary studies, and very few updates led to a change in conclusion. We did not find clear evidence that updates were undertaken faster when new evidence was made available. New approaches for finding early signals that a systematic review conclusion is at risk of change may be useful in allocated resources to the updating of systematic reviews.

Project description:Objective: The aim of this study is to critically appraise whether published systematic reviews/meta-analyses of traditional Chinese medicine for adults with ischemic stroke are of sufficient quality and to rate the quality of evidence using the Grading of Recommendations, Assessment, Development, and Evaluation approach. Method: A literature search was performed in the Cochrane Library, PubMed, Chinese National Knowledge Infrastructure, and SinoMed databases by March 2022. The inclusion criteria were systematic reviews/meta-analyses of traditional Chinese medicine in adults who suffered from ischemic stroke. A Measurement Tool to Access Systematic Reviews 2 (AMSTAR-2) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Abstract (PRISMA-A) statements were used to assess the methodological and reporting quality of the included reviews. The Grading of Recommendations, Assessment, Development, and Evaluation system was utilized to assess each report’s evidence level. Results: Of the 1,908 titles and abstracts, 83 reviews met the inclusion criteria. These studies were published between 2005 and 2022. The results of AMSTAR-2 showed that 51.4% of the items were reported, but the registration, reasons for the inclusion of study design, the list of excluded studies, and funding information were ignored in the majority of the reviews. The results of PRISMA-A showed that 33.9% of items were reported, and the information on registration, limitation, and funding was not available in many publications. The assessment of the evidence with the Grading of Recommendations, Assessment, Development, and Evaluation showed that more than half (52/83) of the included studies had either low or very low levels of evidence. Conclusion: The reporting quality in the abstract of systematic reviews/meta-analyses on traditional Chinese medicine for ischemic stroke is poor and does not facilitate timely access to valid information for clinical practitioners. Although the methodological quality is of a medium level, this evidence lacks certainty, especially with a high risk of bias in individual studies.

Project description:BackgroundSystematic reviews support health systems and clinical decision-making by identifying and summarizing all existing studies on a particular topic. In 2009, a comprehensive description of child-relevant systematic reviews published in the Cochrane Database of Systematic Reviews was compiled. This study aims to provide an update, and to describe these systematic reviews according to their content and methodological approaches.MethodsAll child-relevant systematic reviews published by the Cochrane Collaboration in the Cochrane Database of Systematic Reviews (CDSR) as of March, 2013 were identified and described in relation to their content and methodological approaches. This step equated to an update of the Child Health Field Review Register (CHFRR). The content of the updated CHFRR was compared to the published 2009 CHFRR description regarding clinical and methodological characteristics, using bivariate analyses. As the Cochrane Collaboration has recognized that disease burden should guide research prioritization, we extracted data from the Global and National Burden of Diseases and Injuries Among Children and Adolescents Between 1990 and 2013 study in order to map the distribution of the burden of disease in child health to the distribution of evidence across Review Groups in the CHFRR.ResultsOf the 5,520 potential Cochrane systematic reviews identified, 1,293 (23.4%) were child-relevant (an increase of 24% since 2009). Overall, these reviews included 16,738 primary studies. The most commonly represented Review Groups were Airways (11.5%), Cystic Fibrosis and Genetic Diseases (7.9%), Acute Respiratory Infections (7.8%), Developmental, Psychological and Learning Problems (6.7%), and Infectious Diseases (6.2%). Corresponding authors were most often from Europe (51%), North America (15%), and Australia (15%). The majority of systematic reviews examined pharmacological interventions alone (52% compared to 59% in 2009). Out of 611 reviews that were assessed as up-to-date, GRADE was used in 204 (35%) reviews to assess the overall quality of the evidence, which was often moderate (35.6%) or low (37.8%) for primary outcomes. Ninety percent of reviews that were assessed as up to date used the Cochrane Risk of Bias tool, or a modified version, to assess methodological quality. Most reviews conducted one or more meta-analyses (73%). Among the 25 leading causes of death globally, the Review Groups associated with the largest number of causes were: 1) Infectious Diseases, 2) Anaesthesia, Critical, and Emergency Care, 3) Injuries, 4) Pregnancy and Childbirth (PC), and 5) Neonatal. There were large discrepancies between the number of causes of mortality that each Review Group was associated with and the total amount of evidence each Review Group contributed to the CHFRR. Ninety-eight percent of the causes of mortality in 2013 were from developing nations, but only 224 (17.3%) reviews had corresponding authors from developing countries.ConclusionThe content and methodological characteristics of child-relevant systematic reviews in the Cochrane CHFRR have been described in detail. There were modest advances in methods between 2009 and 2013. Systematic reviews contained in the CDSR offer an important resource for researcher's, clinicians and policy makers by synthesizing an extensive body of primary research. Further content analysis will allow the identification of clinical topics of greatest priority for future systematic reviews in child health.

Dataset Information

Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence.

Publications

Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets